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Abstract 

The  C-17A  Globemaster  III  began  service  on  June  14,  1993  and  demonstrates  some  of 
the  highest  levels  of  aircraft  availability  in  the  USAF  inventory.  The  sustainment 
activities  responsible  for  these  levels  are  conducted  through  a  public-private  partnership. 
The  Air  Force  entered  this  sustainment  partnership  without  the  advantage  a  business  case 
analysis  (BCA).  In  June,  2006,  the  Department  of  Defense  Inspector  General  (DoDIG) 
directed  the  creation  of  a  BCA  for  the  C-17A  program  to  assess  all  alternative 
sustainment  strategies,  along  with  recommendations  for  best-value  alternatives.  Starting 
in  2007  and  completed  in  2009,  the  BCA  provided  analysis  of  three  areas  for  predicting 
sustainment  objective  accomplishment:  benefits,  cost,  and  risk.  While  the  2009  BCA 
encompassed  the  most  detailed  assessment  of  alternatives  to-date,  it  contained  shortfalls. 
Analysis  of  the  benefit  metrics  showed  significant  interaction  between  the  chosen  metrics 
resulting  in  skewed  analysis  and  difficulty  discerning  between  options.  The  objective  of 
this  study  was  to  develop  mutually  exclusive  BCA  benefit  metrics  to  assess  sustainment 
strategies,  weights  to  use  within  a  decision  analysis  framework  to  better  inform  future 
BCAs,  and  develop  business  rules  to  exclude  data  during  transitional  phases  of 
sustainment  in  supporting  datasets. 
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C-17A  SUSTAINMENT  PERFORMANCE  METRICS  ASSESSMENT: 
REPAIR  SOURCE  IMPACT  ON  SUSTAINMENT  FOR  FUTURE  BUSINESS 
CASE  ANALYSIS  DEVELOPMENT 


I.  Introduction 


Background 

In  June,  2006  the  Department  of  Defense  Inspector  General  (DoDIG)  directed  a 
business  case  analysis  (BCA)  be  conducted  on  sustainment  strategies  for  the  C-17A. 
Specifically,  the  DoDIG  stated  the  BCA  should  be  an  objective  analysis  thoroughly 
evaluating  multiple  sustainment  options  for  the  C-17  aircraft  to  ensure  the  government 
makes  a  knowledgeable,  best-value  decision  for  long-term  sustainment  (DoDIG,  2006). 

At  the  time  of  publication  in  2009,  the  C-17A  was  both  in  a  production  and  a  sustainment 
phase  and  was  operated  by  eleven  domestic  and  three  foreign  military  bases  with  slightly 
more  than  206,000  total  flying  hours  (AFTOC,  2013).  The  2009  C-17A  BCA  (hereafter 
referred  to  as  the  2009  BCA)  represented  a  collection  of  data  and  analysis  to  offer  the 
USAF  and  DoD  senior  leaders  several  courses  of  action  for  the  sustainment  options  of  the 
C-17A  with  respect  to  metrics  assessing  benefits,  cost,  and  risk  of  different  sustainment 
options.  The  BCA  was  the  culminating  efforts  from  over  20  people  spanning  2  years  and 
sought  to  produce  recommendations  on  the  30-year  sustainment  program. 

The  C-17A  Globemaster  III  began  service  on  June  14,  1993.  Nearly  20  years  into 
its  service  life,  the  C-17A  demonstrates  some  of  the  highest  levels  of  aircraft  availability 
of  any  aircraft  in  the  USAF  inventory.  Part  of  the  concern  from  the  DoDIG  was  directed 
at  the  value  aspects  of  this  aircraft  availability.  Since  the  most  recent  standard  set  for 
aircraft  availability  in  2009,  the  standard  has  been  routinely  exceeded  but  not  without 
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accompanying  costs.  In  order  to  fully  inform  the  discussion  on  sustainment  strategies, 
we  must  also  be  able  to  quantify,  in  terms  of  cost,  what  exceeding  the  aircraft  availability 
standard  means.  Next,  we  must  determine  whether  this  is  an  appropriate  trade  off  to 
make.  A  BCA  is  an  important  step  in  this  assessment  and  one  to  be  carefully  eonsidered 
and  ereated.  The  2009  C-17A  BCA  reeommendations  positively  affirmed  a  government- 
led  eollaboration  between  the  USAF  and  The  Boeing  Company  (hereafter  referred  to  as 
Boeing)  as  the  best-value  solution  for  C-17  sustainment  (Business  Case,  2009). 
Furthermore,  the  reeommendations  also  positively  contributed  to  the  USAF’s  goal  of 
maximized  aircraft  availability  through  incentives  given  for  performance  and  the 
reduetion  of  unavailability  based  on  maintenance,  supply,  and  depot  issues. 

Overall,  the  2009  BCA  assessed  seven  potential  cases  for  consideration.  The 
eases  were  constructed  according  to  integrator  type  and  support  provider.  Furthermore, 
evaluations  considered  the  cases’  ability  to  attain  certain  specific  and  measureable 
attributes  relating  to  depot  performance.  These  attributes  were  compiled,  then  prioritized 
by  subject  matter  experts  examining  workload  assignments  relative  to  pre-defined 
criterion.  Following  the  analysis,  the  BCA  set  forth  three  potential  courses  of  action, 
recommending  one  case  as  the  best  value  of  benefits,  cost,  and  risk.  From  the 
reeommendation  of  the  BCA,  the  USAF,  aeting  on  behalf  of  the  other  governments  in 
partnership  with  the  C-17A  fleet,  made  several  changes  to  the  recommendation.  Boeing 
was  then  presented  with  the  ehanged  reeommendation.  Boeing  offered  to  meet  the  cost 
savings  requirements  without  transitioning  to  the  reeommended  course  of  action  inclusive 
of  the  ehanges  proposed  from  the  USAF.  The  Boeing  offer  represented  the  status  quo 
with  continued  efforts  to  reduee  costs  and  realize  performance  improvements.  The 
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current  state  of  affairs  notes  little  change  to  the  2009  environment  with  the  exeeption  of 
the  Government  acting  as  the  primary  souree  manager  and  the  ongoing  effort  to  transition 
the  engines. 

The  2009  BCA  analyzed  seven  different  sustainment  eases.  Case  1  represented 
the  present  conditions  at  the  time  of  the  study  and  the  baseline  case  in  whieh  Boeing 
retained  obligations  as  the  Total  System  Support  Responsibility  (TSSR).  Case  2 
represented  a  theoretical  boundary  condition  in  which  Boeing  retains  TSSR,  and,  in 
addition,  all  Government  sustainment  funetions  would  transition  to  eontractor  support 
(Business  Case,  2009).  Case  3  represented  another  theoretieal  boundary  condition  of 
100%  organic  sustainment.  This  case  obligated  Warner  Robins  Air  Logisties  Center 
(WR-ALC)  support  through  a  system  support  manager,  namely,  all  contraetor 
sustainment  functions  transition  to  the  Government.  This  construet  would  have  been 
required  to  undergo  a  formal  contraet  bidding  and  award  process  in  order  for  the  WR- 
ALC  to  assume  this  responsibility.  Case  4  represented  a  Boeing  TSSR/organic  mix 
uneonstrained  by  Core  and  50/50  requirements  (Business  Case,  2009).  This  ease  ealled 
for  a  eontraetor-led  best  value  collaboration  of  contractor  and  government  sustainment 
activities  not  constrained  by  Core  or  50/50  requirements  by  either  requesting  a  waiver  or 
relying  on  separate  weapons  systems  to  balance  Core  and  50/50  requirements  (Business 
Case,  2009).  Case  5  represented  a  Government  Produet  Support  Manager 
(PSM)/Contractor  Mix  and  also  unconstrained  by  Core  and  50/50.  This  case  represented 
a  government-led  best  value  mix  of  contraetor  and  government  sustainment  activities 
(Business  Case,  2009).  Case  6  represented  a  Boeing  TSSR/Organic  Mix,  constrained  by 
Core  and  50/50  requirements  with  a  contraetor-led  best  value  mix  of  contraetor  and 
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Government  sustainment  aetivities.  Case  7  represented  a  Government  PSM/Contraetor 
Mix,  eonstrained  by  Core  and  50/50  requirements  with  a  government-led  best  value  mix 
of  eontraetor  and  government  sustainment  aetivities. 

The  2009  BCA  reeommended  Case  7  as  the  best  eourse  of  aetion  to  the  USAF. 
However,  the  USAF  augmented  this  ease  with  program  management  and  engineering 
responsibility  given  to  the  Government.  The  USAF  briefed  the  augmented  BCA  ease  to 
Boeing  as  the  desired  eourse  of  action  based  on  attained  levels  of  aircraft  availability 
(among  many  other  attributes)  and  depot  performance.  In  lieu  of  making  this  transition, 
Boeing  agreed  to  meet  savings  projections  while  operating  under  the  current  state  (Case 
1)  with  several  changes.  Those  changes  were;  increased  program  management  workload, 
product  support,  materiel  and  depot  management;  and  avionics  and  software 
responsibility  performed  primarily  by  the  contractor  (90%).  Based  on  a  recommendation 
from  the  2009  BCA,  the  USAF  also  started  a  program  for  transitioning  the  engines  to  a 
direct  support  contract  (with  potential  of  savings  at  $1.6B). 

BCA  Shortfalls 

The  2009  BCA  was  the  first  of  its  kind  for  the  C-17A  Globemaster  III.  As  such, 
the  BCA  helped  pioneer  certain  aspects  of  valuable  analysis  an  provided  key  insights  to 
this  project  but  not  without  shortfalls.  The  2009  BCA  had  several  shortfalls  in  the 
analysis  and  the  recommendations.  The  use  of  weighted  dollar  values,  overlapping  and 
inclusive  benefit  metrics,  analogous  weapons  system  data  derived  from  sustainment 
strategies  not  in  use,  incomplete  analysis  of  the  savings  estimate  through  engine 
competition,  no  analysis  of  alternate  cost  savings  methods,  and  recommendations 
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exceeding  the  scope  of  analysis  all  contributed  to  a  product  of  limited  use.  Weighing 
dollar  values  carries  the  implication  certain  dollars  are  more  important  than  other  dollars. 
Overlapping  benefit  metrics  lead  to  double-  and  triple-counting  with  interactions  between 
variables  measuring  the  same  trait.  Weapons  system  data  was  either  derived  or 
extrapolated  for  certain  benefit  metrics  in  the  2009  BCA.  Sustainment  strategies  not  in 
use  are  not  defendable  through  data  or  historical  evidence  were  used  to  make 
recommendations  used  in  the  2009  BCA.  Data  for  the  component  product  deficiency 
rates  were  collected  from  only  three  national  stock  numbers.  The  2009  BCA 
recommendation  to  compete  the  sustainment  contract  for  the  C-17A  engines  is  not  a  best- 
value  recommendation  since  it  did  not  include  analysis  of  benefits,  cost,  and  risk.  The 
2009  BCA  also  did  not  assess  changes  in  incentive  structures.  Consequently,  a  complete 
analysis  was  not  performed. 

One  of  the  BCA  shortfalls  was  the  discrepancy  between  the  analysis  and  the 
scope  of  the  recommendations  set  forth.  The  aim  of  the  2009  BCA  was  to  thoroughly 
assess  sustainment  options  for  the  C-17A  and  ensure  the  Government  makes  a 
knowledgeable,  best-value  decision  for  long-term  sustainment  (Business  Case,  2009). 

The  2009  BCA  used  analysis  of  benefits,  their  respective  costs  projected  out  to  a  term  of 
thirty  years,  and  their  associated  risks  over  that  period  as  described  in  the  various  models 
used  (Business  Case,  2009).  However,  not  all  recommendations  followed  this  informed 
process.  One  recommendation  was  not  a  best-value  recommendation  in  terms  of  the 
associated  risk,  cost,  benefit,  and  incentive  analysis  but  followed  with  the  estimate  of 
significant  cost  savings  to  the  C-17A  program. 
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The  BCA  attempted  to  ereate  unique  sustainment  eases  to  attain  the  best-value 
mix  or  performance,  cost,  and  risk  for  the  30-year  cost  profile.  The  2009  BCA  cost 
profile  did  not  include  analysis  of  cost  changes  over  the  lifecycle  due  to  the  reliability 
curve  (bathtub  effect),  aging  aircraft,  changes  in  fleet  size,  and  changes  to  the  operational 
environment.  No  sensitivity  analysis  was  not  performed  around  these  potential  impacts 
to  generate  a  range  of  suspected  values  given  a  future  state.  Lastly,  the  unique 
sustainment  cases  did  not  correspond  to  sustainment  programs  used  by  other  aircraft. 
These  derived  data  profiles  did  not  have  fully  listed  assumptions. 

The  2009  BCA  did  not  propose  different  contractual  incentive  structures  the 
Government  may  use  as  options  to  incentivize  Boeing  for  sustainment  activities  in 
addition  to  the  proposed  sustainment  cases.  In  other  words,  the  2009  BCA  aimed  to 
show  the  only  method  of  improving  sustainment  or  achieving  certain  sustainment  goals 
was  to  integrate  government  and  contractor  activities  without  looking  at  a  different 
potential  alternative  altogether  -  a  set  of  different  incentive  structures  for  the  contractor 
in  contrast  to  the  current  incentive  options.  Incentive  analysis  seeks  to  make  changes 
within  contracts  to  make  them  cheaper  by  considering  reductions  in  performance 
thresholds,  changes  to  incentives  offered,  or  changes  to  the  entire  payment  contract  used. 

Additionally,  the  2009  BCA  used  weighted  dollar  values  to  assess  projected 
values  of  sustainment  cases.  Weighted  dollar  values  can  be  misleading  by  placing  higher 
priorities  on  certain  dollars  over  others.  Dollar- to-dollar  comparisons  are  one  tool  for 
comparison;  weighing  dollar  values  complicates  comparisons.  Lastly,  the  2009  BCA 
stated  the  largest  cost  savings  could  be  realized  through  the  competition  of  the 
sustainment  contract  for  the  C-17A  engines  (Business  Case,  2009). 
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The  BCA  demonstrated  shortfalls  insofar  as  the  interaetions  between  metrics 


chosen.  A  major  point  in  decision  analysis  is  to  use  decision  variables  that  do  not 
measure  the  same  attribute.  This  key  aspect  of  the  2009  BCA  represents  the  primary 
focus  of  this  project.  Aircraft  availability  represented  the  highest  weighted  metric. 
However,  the  sub-factors  of  aircraft  availability  were  also  metrics  used  in  the  analysis  of 
depot  sustainment.  The  presence  of  both  aircraft  availability  and  its  sub-factors  obscure 
clearly  differentiated  options  since  components  of  one  factor  are  included  again  in  others. 
Choosing  one  option  over  another  is  complicated  when  one  of  the  options  includes 
factors  of  the  second  option  -  the  two  options  are  not  distinctly  different  from  each  other 
because  the  first  directly  influences  the  second  and  vice  versa.  This  project  sought  to 
propose  metrics  without  such  interactions  and  to  propose  a  framework  for  their  use  to 
assess  and  to  predict  the  impact  of  changes  in  the  sources  of  sustainment.  This  project 
uses  sustainment  structures  currently  used  in  similar-type  aircraft  to  assess  their  efficacy 
and  performance  with  regard  to  aircraft  availability. 


Figure  1:  Aircraft  Availability  Sub-Factors 
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Instead  of  using  primary- factor  and  sub-factor  metrics,  this  study  uses  only  sub-factor 
metrics  to  prevent  such  interactions  between  the  chosen  metrics.  Mutually  exclusive 
metrics  allow  decision  assessment  based  on  their  accomplishment  of  desired  objectives. 

The  original  scope  of  this  project  was  to  update  the  2009  BCA  with  current 
performance  data  relevant  to  the  changes  in  the  operating  environment,  fleet  size, 
associated  sustainment  risks,  and  their  respective  costs.  The  first  step  in  the  original 
scope  was  to  evaluate  the  assumptions  of  the  model  and  determine  whether  they  were  still 
valid.  However,  under  a  review  process  intended  to  produce  an  updated  version  of  the 
2009  BCA  to  meet  with  the  five-year  requirement  for  program  review,  analysis  of  the 
benefit  metrics  showed  the  aforementioned  sub-factor  interference.  The  benefit  metrics 
did  not  all  represent  clear  alternatives  making  the  choice  between  the  proposed 
sustainment  strategies  unclear.  The  intent  of  decision-making  is  to  produce  a  desired 
consequence.  However,  when  there  is  little  difference  between  the  alternatives,  one 
cannot  be  sure  the  decision  made  will  have  the  desired  consequences. 

The  2009  BCA  Benefit  Metrics  formed  the  foundation  of  analysis  from  which 
costs  and  risks  were  then  derived.  Four  of  the  seven  Benefit  Metrics  assigned  to  identify 
differences  in  performance  measures  corresponding  to  sources  of  repair  measured  sub¬ 
factors  of  the  primary  metric  -  aircraft  availability.  Consequently,  when  data  for  one 
benefit  metric  was  collected  and  input  into  the  benefit  model,  it  already  included  the  data 
for  other  metrics  subsequently  input  into  the  model. 
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Sub-factors  of 
Aircraft  Availability 


Figure  2:  2009  BCA  Benefit  Metrics  Sub-Factors  of  Aircraft  Availability 

As  a  result,  metrics  were  double-counted.  If  sustainment  decisions  were  designed  to 
improve  stockage  effectiveness  rates,  for  example,  that  decision  would  not  be  different 
from  a  decision  to  improve  non-mission  capable  rates  for  supply,  customer  wait  times,  or 
aircraft  availability.  The  consequences  for  the  decision  are  not  different  because  the 
decision  metrics  were  not  mutually  exclusive. 

The  measurement  of  sub-factors  in  addition  to  the  primary  metric  introduced 
analysis  problems  by  giving  the  sub-factors  more  importance,  double-measuring,  and 
failing  to  distinguish  between  viable  alternatives.  Consequently,  the  BCA 
recommendations  did  not  provide  accurate  measure  of  each  of  the  seven  cases  and  an 
update  to  the  2009  BCA  would  have  resulted  in  further  skewed  analysis  and  the  inability 
to  delineate  between  alternatives.  Since  the  release  of  the  2009  BCA,  there  has  been 
interest  in  re-assessing  the  model  and  assumptions  to  more  realistically  incorporate 
mutually  exclusive  variables  without  sub-factor  interaction. 

Additional  shortfalls  of  the  BCA  related  to  areas  outside  the  scope  of  this  project 
are  included  to  better  inform  future  BCAs.  In  the  2009  BCA,  there  were  unclear 
descriptions  on  the  methodology  of  weighed  dollar  values  used  in  the  cost  analysis  of  the 
models  and  a  lack  of  projected  ranges  for  each  of  the  seven  cases.  The  metrics  chosen  in 
the  benefits  section  were  the  amalgamation  of  many  surveyed  responses  from  both  the 


Prioritized  Benefit  Attributes 

Weighted  Value 

Normalized  Aircraft  Availability  Rates 

24.0% 

On-Time  Delivery  Rate 

20.2% 

Routine  Depot  Assistance  Request  Response  Time 

13.5% 

Indexed  Non-Mission  Capable  Supply  Rate 

Ci2.9%'y^  ^ 

Component  Product  Quality  Deficiency  Report  Rate 

C  8.2%  3^ // 

Customer  Wait  Duration 

c;7.9%>'// 

Issue  Effectiveness  Rate 

C  7.2%  ^  / 

Stock  Effectiveness  Rate 

c' B.0% 
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field  and  program  experts.  Ultimately,  these  metries  failed  to  refieet  the  items  of 
influence  on  the  balanced  score  card  (BSC)  used  by  Air  Force  Materiel  Command 
(AFMC).  The  “On-time  Delivery  Rate  (original  and  revised)”  metrics  were  determined 
to  have  an  additional  weighting  of  53%  and  47%.  It  is  unclear  how  this  method  best 
informs  the  weighted  utility  score  of  those  values.  Analysis  performed  for  this  research 
project  could  not  conclude  the  modcFs  purpose  for  this  step. 

Problem  Statement 

This  graduate  research  project  (GRP)  seeks  to  develop  a  defensible  benefit 
analysis  methodology  for  the  C-17A  BCA.  The  methodology  establishes  mutually 
exclusive  benefit  metrics  to  assess  depot  performance  within  existing  sources  of  repair; 
proposes  a  structure  for  weights  and  comparisons  within  a  decision  analysis  framework  to 
better  inform  future  analysis;  and  establishes  business  rules  for  data  that  excludes  data 
during  sustainment  transitional  periods.  Since  analogous  weapons  systems  and  their 
corresponding  sustainment  strategies  established  the  framework  for  comparisons  in  this 
study,  organizational  structures  to  increase  aircraft  availability  may  also  be  inferred  to 
provide  inputs  for  future  BCAs  and  develop  a  framework  for  decision  analysis  for  use 
across  other  weapons  systems. 

Research  Objectives 

The  purpose  of  this  project  is  two-fold:  to  develop  mutually  exclusive  BCA 
benefit  metrics  to  assess  sustainment  strategies  and  to  develop  a  model  to  use  within  a 
decision  analysis  framework  to  inform  future  BCAs. 
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Research  Questions 

Research  Question  1:  Are  the  2009  BCA  prioritized  benefits  metrics  mutually  exclusive? 
Research  Question  2:  How  were  the  weights  determined? 

Research  Question  3:  Is  there  evidence  this  is  the  correct  weighing  of  these  benefit 
metrics? 

Research  Question  4:  What  metrics  can  best  capture  the  impact  on  aircraft  availability 
given  changes  in  the  source  of  repair? 

Research  Question  5:  What  sources  of  data  exist  for  the  assessment  of  sustainment 
performance  strategies? 

Assumptions 

Assumption  1:  The  main  factors  affecting  aircraft  sustainment  are  addressed  through 
policies  directly  relating  to  non-mission  capable  for  supply  (NMCS)  rate,  non-mission 
capable  for  maintenance  (NMCM)  rate,  non-mission  capable  for  both  supply  and 
maintenance  (NMCB)  rate,  and  Depot  Time  rate. 

Assumption  2:  The  data  collected  on  chosen  metrics  is  accurate  and  complete,  reflecting 
actual  rates  of  the  chosen  aircraft  availability  sub-factors. 

Assumption  3:  The  metrics  chosen  reflect  the  most  important  factors  of  aircraft 
availability,  and  furthermore,  these  metrics  impact  sustainment  decisions. 

Assumption  4:  There  are  inherent  differences  between  NMCS,  NMCM,  NMCB,  and 
Depot  Time  rates  based  on  the  source  of  repair. 

Assumption  5:  The  average  values  of  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates  will 
yield  the  best  and  most  logical  value  for  those  metric  over  the  specified  duration  of  time. 
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Assumption  6:  Measuring  sub-factors  of  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates 
result  in  the  best  assessment  of  the  source  of  repair’s  impact  of  these  sub-factors. 
Assumption  7:  Past  performance  is  capable  of  predicting  average  values  of  NMCS, 
NMCM,  and  NMCB  to  project  future  values  given  a  constant  source  of  repair. 
Assumption  8:  Data  for  first  three  years  of  aircraft  sustainment  after  a  change  in  repair 
source  is  not  representative  of  the  eventual  sustainment  level  achieved  along  the 
reliability  growth  curve  once  the  aircraft  reaches  maturity. 

Assumption  9:  Data  for  first  five  years  of  aircraft  sustainment  after  initial  production  is 
not  representative  of  the  eventual  sustainment  level  achieved  once  the  aircraft  reaches 
maturity  along  the  reliability  growth  curve. 

Business  Rules 

Business  rules  created  for  this  project  allowed  for  consistent  and  appropriate  data 
use  representative  of  sustainment  strategies  consistently  and  appropriately.  The  business 
rules  were  guides  to  aid  in  the  comparison  of  NMCS,  NMCM,  NMCB,  and  Depot  Time 
rates  across  the  various  sustainment  strategies  evaluated.  Business  rules  addressed  which 
models  of  aircraft  could  be  included  in  the  analysis,  which  years  of  data  were  appropriate 
representations  of  sustainment  strategies,  and  which  sustainment  strategies  would  be 
assessed. 

Sustainment  programs  currently  in  use  were  the  basis  of  comparison  for  this 
project.  The  study  used  data  from  aircraft  availability  to  represent  sustainment  programs 
currently  in  use.  This  project  does  not  seek  to  fully  capture  every  aspect  of  aircraft 
availability  but  instead  measure  the  causal  relationships  of  activities  within  the 
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sustainment  programs  studied  as  a  method  of  proposing  a  similar  sustainment  strategy  for 
the  C-17A.  Coneurrently,  the  projeet  is  limited  to  the  C-17A  since  the  weighted  values 
used  in  the  model  were  informed  by  the  2009  BCA. 

Aircraft  availability  includes  non-mission  capable  statuses  for  maintenance 
(NMCM),  supply  (NMCS),  both  maintenance  and  supply  (NMCB),  and  unit  possessed 
not  reported  (UPNR).  Data  for  unit  possessed  not  reported  (UPNR)  was  not  included  in 
the  measures  of  depot  effectiveness.  Data  for  UPRN  were  excluded  because  the  metric 
was  a  measurement  of  only  unit-level  impact  of  downtime  and  did  not  represent  a  metric 
with  impact  beginning  on  the  depot  level.  Additionally,  UPNR  represents  a  short-term 
status  for  an  aircraft  awaiting  disposition  -  that  is,  the  aircraft  is  not  mission  capable  but 
the  exact  nature,  classification  code,  or  reason  the  aircraft  is  broken  has  not  been 
officially  designated.  Aircraft  in  a  UPNR  status  do  not  stay  in  this  status  for  long  periods 
of  time  and  eventually  get  assigned  a  more  appropriate  status  of  NMCM  or  NMCS. 
Insofar  as  their  impact  to  aircraft  availability,  UPNR  metrics  represent  aircraft  with 
downtime  that  is  not  charged  to  the  unit  with  maintenance  responsibility.  UPNR  is  not 
directly  tied  to  depot-level  maintenance  impact.  To  prevent  sub-factor  interactions, 
PDQR  and  UPNR  were  removed  from  the  benefit  metrics  assessed. 

Analyses  of  only  the  main  sub-factors  of  aircraft  availability  are  certain  to  leave 
some  margins  unexplored.  There  are  many  areas  and  separate  metrics  that  impact 
NMCS,  NMCM,  NMCB,  and  Depot  Time.  Customer  wait  time,  issue  effectiveness  rate, 
and  stockage  effectiveness  rate  are  all  measures  impacting  NMCS.  This  research  aims  to 
differentiate  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates  by  source  of  repair  to  further 
inform  analysis  of  tertiary  or  subordinate  metrics  as  areas  to  concentrate  efforts. 
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Specifically  noting  the  over-arching  metric,  such  as  NMCS,  is  certain  to  under-value 
policies  and  strategies  that  target  the  more-immediate  and  measurable  factors  that 
contribute  to  NMCS.  In  other  words,  all  activities  associated  with  reducing  NMCS, 
NMCM,  NMCB,  and  Depot  Time  are  not  ubiquitous  across  all  sustainment  programs. 
Furthermore,  some  aircraft  are  more  complex  than  others  and  consequently  result  in 
differing  aircraft  availability  sub-factor  rates,  regardless  of  sustainment  strategy.  By 
choosing  to  restrict  the  benefit  metrics  of  this  project  to  the  lagging  factors,  we  are 
limited  to  measuring  their  importance  and  not  all  of  the  smaller  components  that  go  into 
the  larger  metrics. 

There  was  insufficient  time  and  study  devoted  to  the  precise  and  relevant  weights 
of  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates  used  in  this  model.  As  these  metrics 
posed  the  most  logical  metrics  and  given  their  mutually  exclusive  nature,  they  may  not 
represent  the  only  method  of  assessing  sustainment  program  performance  benefits. 
Further  analysis  and  study  should  be  devoted  to  both  the  efficacy  of  their  use  and  the 
respective  weights  assessed  to  each  metric.  Of  the  potential  benefit  metrics  proposed  for 
inclusion  in  this  project,  those  metrics:  product  quality  deficiency  report  rates,  mean  time 
between  repairs,  and  maintenance  man-hours  were  not  chosen.  They  were  excluded  from 
the  benefit  model  in  this  project  because  their  inclusion  either  did  not  provide  sufficient 
trade  space  or  their  impact  was  already  captured  in  the  chosen  metrics. 

An  additional  business  rule  related  to  which  aircraft  sustainment  program  data 
were  entirely  excluded  from  the  analysis.  Aircraft  models  completely  phased  out  with 
sustainment  systems  no  longer  in  place  were  not  used.  The  E-8A  was  not  used  since  its 
last  recorded  presence  in  the  primary  database  used  for  this  project  was  1995  (LIMS-EV, 
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2013).  The  C-5C  was  not  used  sinee  there  are  only  two  of  these  aireraft  in  the  USAF 
inventory  and  any  analysis  of  their  sustainment  strategy  using  the  metries  proposed  in 
this  study  would  have  dramatically  skewed  the  data  (LIMS-EV,  2013).  The  C-130A  and 
B  models  were  not  used  because  they  were  phased  out  in  1994  and  1995,  respectively 
(LlMS-EV,  2013).  Sustainment  program  data  for  the  KC-135A,  B,  and  E  models  were 
also  not  used.  The  KC-135A  and  B  models  have  been  converted  to  R  and  T  models  over 
the  years  (KC-135,  2011).  The  KC-135E  was  retired  in  2009  and  data  on  the  sustainment 
of  this  aircraft  was  not  collected  (KC-135,  2011). 

Other  data  excluded  from  this  analysis  pertained  to  more  recent  procurements  in 
the  USAE  inventory.  Data  on  the  C-5M  and  C-130J  sustainment  programs  were 
collected.  The  C-5M  showed  NMCM  rates  fluctuating  between  18%  and  25%,  evidence 
of  an  immature  reliability  growth  curve  in  addition  to  only  having  9  aircraft  in  the 
inventory.  The  C-130J  sustainment  program  appeared  to  represent  not  only  a  competitive 
sustainment  strategy  but  would  have  resulted  in  a  lower  overall  benefit  model  score  than 
the  current  C-17  sustainment  strategy.  The  reason  the  program  was  excluded  was 
because  of  the  recent  opening  of  the  dedicated  C-130J  Depot  at  Robins  Air  Eorce  Base, 
Georgia.  Based  on  the  first  time  use  of  a  new  maintenance  and  sustainment  strategy 
similar  to  ones  commonly  used  in  airlines,  the  program  is  still  in  the  transitory  period. 
There  is  still  learning  that  has  not  yet  been  reflected  in  sustainment  performance  factors. 

The  exclusion  of  data  for  the  C-5M  and  C-130J  sustainment  strategies  represents 
another  limitation  of  this  project.  The  recent  nature  of  operations  at  the  C-130J  Depot  led 
to  the  exclusion  of  that  data  all-together.  Based  on  the  design  of  sustainment  operations 
at  the  C130J  Depot  and  the  unique  apportionment  of  only  those  activities  that  are  best 
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suited  for  execution  at  the  depot  may  likely  result  in  low,  long-term  Depot  Time  rates. 

Of  note,  the  program  data  for  the  C-130J  was  available  for  analysis  in  LIMS-EV.  Depot 
Time  rates  were  recorded  for  the  period  of  time  since  its  initial  production.  However,  the 
absence  of  a  dedicated  depot  resulted  in  the  likelihood  those  rates  were  not  comparable  to 
other  aircraft’s  depot  time  rates.  Consequently,  all  data  for  the  C-130J  program  was 
excluded.  Additionally,  the  C-5M  was  found  to  be  in  the  immature  stage  of  its  expected 
reliability  growth  curve  by  this  project.  There  is  evidence  to  demonstrate  the  current  C- 
5M  and  C-130J  sustainment  strategies  are  effective.  However,  the  newness  of  the 
airframe  resulted  in  exclusion  to  account  for  learning,  training  for  maintainers,  and 
development  and  maturation  of  logistics  and  supply  chain  functions.  All  these  factors 
have  shown  to  improve  performance  with  maturity  but  a  fair  assessment  of  either 
aircraft’s  sustainment  strategy  cannot  be  made  at  this  time  with  regard  to  NMCS, 

NMCM,  NMCB,  and  Depot  Time  rates. 

Data  considered  was  analyzed  for  anomalies  that  may  skew  analysis  and  vetted 
for  the  impact  of  program  transitions,  maturity  of  the  reliability  curve,  and  number  of 
aircraft  in  the  fleet.  Data  previous  to  FY1994  was  considered  either  too  dated  or 
unrepresentative  of  the  current  practices  in  the  view  of  the  authors  of  this  project. 

Larger  fleet  sizes  create  economies  of  scale  and  lower  values  in  aircraft 
availability  sub-factors  per  individual  aircraft.  It  was  noted  in  the  data  for  aircraft  types 
with  smaller  fleets  the  increase  in  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates  were 
impacted  more  significantly  than  aircraft  types  with  larger  fleets.  There  was  no 
manipulation  or  adjustment  to  the  data  to  account  for  or  correct  this.  Small  fleet  sizes 
dramatically  impact  the  value  of  the  sustainment  performance  metrics  chosen  for  this 
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project.  The  converse  may  also  be  true  whereby  large  fleet  sizes  could  act  to  mask 
certain  inefflciencies  or  spikes  in  sustainment  performance  metrics. 

Lastly,  the  flndings  from  this  project  are  only  relevant  to  the  chosen  beneflt 
metrics.  There  was  no  cost  or  risk  analysis  performed  on  any  of  these  options  and 
therefore,  the  best-value  sustainment  option  cannot  be  determined  from  this  project.  At 
the  conclusion  of  cost  and  risk  assessments,  the  data  may  reveal  the  PSM/PSI 
sustainment  strategy  does  not  represent  the  best-value  option  because  other  sustainment 
strategies  perform  better  in  the  cost  or  risk  areas.  Further  analysis  may  also  reveal  the 
best-value  sustainment  strategy  can  be  achieved  through  changing  how  incentives  are 
offered  to  the  contractor  and  not  embodied  in  a  government-contractor  activities  mix. 
The  degree  and  structure  of  these  incentives  were  not  analyzed  in  this  project  and  are 
mentioned  only  to  strongly  urge  their  inclusion  in  the  future  BCA.  The  full  scope  of 
activities  mix  and  incentives  would  meet  the  intent  to  assess  all  relevant  sustainment 
options  and  not  limit  analysis  to  a  mix  of  government  and  contractor  activities. 

II,  Literature  Review 


Overview 

The  focus  of  the  literature  review  will  be  on  the  need  for  a  fully  informed  BCA, 
the  current  sustainment  strategy  for  the  C-17,  the  methods  of  ascertaining  pertinent  and 
measurable  metrics  to  assess,  a  framework  for  their  analysis,  and  lastly,  the  efflcacy 
behind  this  logical  approach.  Additional  literature  review  will  provide  critique  and 
criticism  to  the  methods  used  in  this  project. 
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The  C-17A  Globemaster  III  began  production  in  June,  1991  and  the  first  aircraft 
was  delivered  in  September,  1993  (Air  Force  Factsheet,  2012).  The  C-17A  has 
undergone  two  distinct  sustainment  program  transitions.  During  its  first  years  of 
production  and  delivery  from  1991  until  1998  it  was  produced  and  sustained  by  Boeing 
(C-17  GISP,  2012).  After  1998,  Boeing  transitioned  to  a  system-level,  performance- 
based  program  using  the  facilities  at  the  Warner  Robins  -  Air  Logistics  Center  (WR- 
ALC)  in  Georgia  (C-17  GISP,  2012).  From  2009  to  the  present,  Boeing  has  acted  as  the 
sole-source  lifecycle  support  provider  (product  support  integrator  or  PSI)  with  the  USAF 
acting  as  the  PSM  charged  with  program  management  among  other  responsibilities. 

The  aircraft  routinely  operates  over  200,000  flying  hours  per  year  and  conducts 
missions  on  all  seven  continents  in  fulfillment  of  the  United  States’  global  obligations 
(AFTOC,  2013).  Missions  vary  from  theater-direct  delivery  to  strategic  airlift, 
aeromedical  evacuation  to  night  airdrop  missions.  It  is  clear  the  C-17  helps  our  nation 
attain  its  national  security  objectives.  Sustaining  this  aircraft  to  ensure  its  availability  and 
viability  for  the  coming  decades  is  of  the  upmost  importance,  perhaps  even  more  so  in 
fiscally  challenging  environments.  A  well-informed  and  carefully  constructed  BCA  will 
undoubtedly  help  toward  those  aims. 

The  Need  for  a  BCA 

A  report  on  public-private  partnerships  at  USAF  maintenance  depots  revealed  a 
lack  of  business  case  analysis-based  support  on  the  efficacy  of  such  partnerships.  The  C- 
17A  sustainment  program  was  one  of  many  sustainment  programs  underway  without  the 
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advantage  of  being  informed  by  a  BCA.  Speeifieally,  the  DoDIG  report  on  July  26, 

2006  determined  the  following: 

Air  Foree  offieials  did  not  use  an  appropriate  methodology  for  making  the 
aequisition  deeision  to  procure  contractor  total  system  support  for  the  C- 
17  aircraft.  Specifically,  the  Air  Force  decision  to  award  total  system 
support  responsibility  was  not  based  on  a  BCA.  This  occurred  because 
senior  Air  Force  officials  directed  the  C-17  program  office  to  focus  efforts 
solely  on  a  partnership  with  the  contractor  without  fully  considering 
additional  sustainment  strategies.  As  a  result,  the  Air  Force  awarded  an 
$871  million  long-term  contract  (with  a  potential  value  of  almost  $5 
billion)  without  proper  and  necessary  support  and  did  not  make  fully 
informed  sustainment  strategy  decisions.  These  decisions  will  impact 
future  options  for  sustaining  the  C-17  when  aircraft  production  is 
complete.  Furthermore,  unless  the  Air  Force  develops  and  completes  a 
thorough  BCA,  it  will  increase  the  risk  of  implementing  for  the  life  of  the 
aircraft  a  sustainment  strategy  that  does  not  achieve  best  value. 

DoDIG,  2006 

A  well-informed  BCA  with  mutually  exclusive  decision  criteria  assessing  both 
contractor-government  activities  mix  and  incentive  programs  would  allow  sufficient 
analysis  of  all  alternatives  to  ensure  the  USAF  received  the  best  value. 

AFI  65-509  details  the  composition  and  components  of  a  BCA.  The  accepted 
definition  of  a  BCA  is: 

A  business  case  analysis  (BCA),  also  referred  to  as  a  business  case  or 
business  plan,  is  a  decision  support  document  that  identifies  alternatives 
and  presents  business,  economic,  risk,  and  technical  arguments  for 
selecting  an  alternative  to  achieve  organizational  or  functional  missions  or 
goals.  BCAs  do  not  replace  the  judgment  of  the  decision  maker,  but  rather 
aid  that  judgment  by  considering  possible  alternatives,  their  costs, 
benefits,  and  risks,  and  the  degree  to  which  they  meet  program  objectives, 
or  are  either  within  budget  constraints  or  require  additional  funding.  A 
BCA  can  vary  in  size  and  scope  depending  on  the  requirements  of  the 
decision  maker  or  reviewing  organization.  The  purpose  of  this  instruction 
is  to  illustrate  what  a  BCA  is  by  comparing  it  to  other  analytical  products, 
explain  when  BCAs  are  required  in  the  Air  Force,  advise  on  when  they 
may  be  completed  even  if  not  strictly  required,  state  the  responsibilities 
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of  offices  involved  in  eompleting  a  BCA,  and  refer  individuals  to 
additional,  detailed  guidanee  on  how  to  aeoomplish  BCAs  (API  65-509, 
2008). 

As  stated  above,  the  BCA  should  consider  the  appropriate  balanee  and  mix  of 
Government  and  contractor  sustainment  activities  but  also  include  finaneial  evaluations 
of  incentive  struetures  in  order  to  comply  with  DoDI  415 1 .21  (2007)  guidance  to 
consider  the  best  business  praetiees. 

The  Current  Sustainment  Strategy 

Currently,  Boeing  exeeutes  PSI  requirements  using  the  facilities,  expertise,  and 
personnel  at  the  Wamer-Robins  Air  Logisties  Center  (WR-ALC).  The  Air  Foree  Life 
Cycle  Management  Center  (AFLCMC)  provides  program  management  and  engineering 
support  to  the  C-17A  as  the  PSM.  The  ultimate  effeet  of  this  unique  partnership  results 
in  some  of  the  lowest  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates  for  any  similar 
aireraft  in  the  USAF  fleet  (LIMS-EV,  2013).  Graphieal  depletions  of  these  rates 
eompared  to  the  rates  of  other  aircraft  are  found  in  Appendix  A. 

There  have  been  numerous  studies  assessing  the  performance  of  depot 
maintenanee  and  sustainment  programs.  Kem  (1999)  PhD  deseribed  the  principle 
performanee  metries  of  depots  should  foeus  on  out-put,  cost,  and  quality.  Furthermore, 
he  advoeated  workload,  efficiency,  effectiveness,  and  produetivity  measures  that, 
aceording  to  Ammons  (1996),  measure  planning  and  budgeting,  operational 
improvements,  resource  allocation,  and  monitoring  tasks.  There  are  a  myriad  of  sourees 
rooted  in  Air  Force  Instructions  (AFI)  and  manuals  directly  describing  how  certain 
activities  will  be  condueted  at  depot  maintenance  faeilities.  In  addition  to  the  standard 
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definitions  and  depot  level  requirements  and  the  ealculation  methods  described  in  API 
21-103,  Equipment  Inventory,  Status  and  Utilization  Reporting,  the  Air  Force  Logistics 
Management  Agency’s  Maintenance  Metrics  handbook  contains  much  of  the  reasoning 
behind  the  principles  used  as  well  as  a  short  history  of  the  evolution  toward  more 
accurate  metrics.  Furthermore,  the  Maintenance  Metrics  handbook  describes  leading  and 
lagging  indicators  of  aircraft  availability  sub-factors  and  directs  action  toward  specific 
responses  in  order  to  affect  positive  responses.  There  is  a  cause  and  effect  relationship 
between  leading  and  lagging  indicators.  As  the  Maintenance  Metrics  handbook  states, 
leading  indicators  directly  reflect  maintenance’s  capability  to  provide  resources  for 
mission  execution  and  lagging  indicators  show  trends  in  those  capabilities  (Maintenance 
Metrics,  2009). 

Metrics 

When  data  was  unavailable  for  the  C-17A,  data  from  analogous  weapons  systems 
were  used.  Table  4-5  in  the  2009  C-17  BCA,  Appendix  4:  Cost  Benefit  Analysis, 
(Business  Case)  shows  which  analogous  weapons  systems  were  used  for  each  particular 
benefit  metric  where  appropriate.  This  study  found  disruptions  to  sustainment 
performance  factors  caused  by  certain  events.  Not  all  metric  data  collected  represented 
the  sustainment  strategy.  When  programs  underwent  transitions,  plateaus  appeared  in 
aircraft  availability  sub-factors.  There  is  no  evidence  to  suggest  data  was  excluded  from 
analogous  weapons  systems  in  the  2009  BCA.  to  demonstrate  the  presence  of 

Troughs  in  aircraft  availability  were  observed  when  the  source  of  a  sustainment 
program  was  changed  or  transitioned  from  the  government  to  a  contractor.  The  range  of 
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KC-135R  data  used  for  the  2009  BCA  was  not  described  in  the  study.  There  is  reason  to 
believe  the  metrics  from  the  2009  BCA  exhibited  similar  traits  during  program 
transitions.  In  addition,  the  partition  of  the  KC-135R  sustainment  program  between 
organic  and  contractor  was  not  fully  described  and  it  is  unknown  what  selection  criteria 
were  used  to  enter  aircraft  into  the  organic  depot  or  the  contractor  depot.  Over  the 
service  life  of  the  KC-135R,  there  have  been  changes  to  its  sustainment  program  and, 
according  to  a  finding  of  this  study,  there  should  be  associated  exclusionary  periods 
during  which  time  this  data  is  not  used.  Lastly,  the  benefit  metric  relying  on  data  from 
analogous  weapons  systems  (component  product  quality  deficiency  report  rate)  was 
based  on  only  three  national  stock  numbers  for  the  C-17A  and  only  eight  from  other 
weapons  systems.  Nonetheless,  this  metric  received  8.2%  of  the  model’s  weighted 
priority  -  roughly  one  third  of  the  priority  of  the  aircraft  availability  rate. 

The  2009  BCA  Current  State  lists  the  assumptions  surrounding  the  operating 
environment  present  at  the  time  of  the  study.  However,  there  is  no  sensitivity  analysis 
present  in  what  impact  additional  C-17A’s  would  have  on  the  metrics  chosen. 
Furthermore,  the  largest  area  of  potential  cost  savings  came  from  assuming  a  cost  savings 
of  $1 .6B  could  be  realized  by  competing  the  sustainment  policy  of  the  engine. 
Throughout  the  report,  there  is  no  evidence  offered  to  support  this  statement  of  savings 
by  openly  competing  the  C-17A’s  Pratt  &  Whitney  FI  17-PW-lOO  turbofan  engines.  The 
cost  of  programs  to  research  and  prepare  for  this  transition  as  well  as  the  potential  risks 
and  benefits  for  such  a  course  of  action  were  not  mentioned  in  the  2009  BCA. 

Depot  assistance  request  response  rate  data  were  divided  into  two  different 
categories  for  reporting  and  recording;  routine,  and  emergency  data.  The  emergency  data 
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for  this  metric  was  unused  because  it  was  not  collected  or  reported  in  a  standardized 
manner  allowing  for  equitable  comparisons  across  weapons  systems.  There  was  no 
evidence  suggesting  routine  data  was  representative  of  this  measured  rate  void  of  the 
entire  body  of  both  emergency  and  routine  requests. 

Such  an  undertaking  as  this  BCA  was,  it  represented  a  valuable  step  toward 
understanding  sustainment  decisions.  Moving  forward  for  the  work  of  countless 
professionals  and  their  valued  inputs,  constructing  the  next  BCA  should  include  lessons 
learned  from  the  2009  BCA  to  improve  its  quality  and  efficacy. 

The  current  C-17A  standards  for  NMCM,  NMCS,  NMCB,  and  Depot  Time  rates 
are  found  in  Figure  9,  Appendix  A.  As  the  program  continues  to  mature,  these  rates  will 
likely  change  representing  new  practices,  strategies  or  policies  and  continued  learning. 
The  current  standards  were  established  in  2009  (LIMS-EV,  2013)  but  have  changed 
many  times  before  then.  There  is  reason  to  believe,  according  to  Wilhelm,  Behrens  and 
Cameron  (2011),  constant  standards  of  NMCM,  NMCS,  NMCB,  and  Depot  Time  do  not 
offer  the  best  solution  to  evaluate  the  capabilities  of  depots.  As  valuable  as  these 
standards  are  to  identifying  performance,  they  do  not  capture  the  full  scope  of  the 
dynamic  activities  present  in  sustainment  strategies.  These  performance  metrics  do  not 
quantify  the  impacts  of  proper  supply  chain  management  activities,  logistics  functions, 
and  many  other  critical  aspects  of  sustainment.  While  these  proposed  metrics  ultimately 
represent  the  acquisition  framework,  events,  and  milestones,  their  effect  is  recognized 
within  the  aggregate  of  their  impact  to  aircraft  availability.  Certainly,  the  trade-space 
with  the  non-mission  capable  sub-factor  rates  and  Depot  Time  rates  represent  the  area  for 
sustainment  focus,  strategy  development,  and  execution.  Fixating  on  certain  standards. 
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according  to  the  Maintenance  Metrics  handbook,  could  result  in  lost  opportunities  to 
make  proportionately  more  progress  towards  positively  affecting  one  metric,  even  to  the 
slight  detriment  of  another  metric  (Maintenance  Metrics,  2009).  The  sub-factors  of 
aircraft  availability  used  for  this  projeet  were  chosen  based  on  their  mutually  exclusive 
nature,  among  other  traits,  and  the  fact  they  can  be  impacted  by  numerous  activities.  The 
standard  applied  to  aircraft  availability  in  the  2009  BCA  did  not  account  for  the  natural 
process  of  learning  over  time.  Furthermore,  program  transitions  causing  sustainment 
performance  disruptions  would  have  been  ineluded  in  this  metrie  as  well  and  eould  have 
skewed  the  analysis. 

BCAs  focusing  on  depot  maintenance  and  aircraft  sustainment  have  historically 
measured  distinct  attributes  and  metries.  Most  applicably,  a  reeognized  “best  practice”, 
as  advoeated  by  Kaplan  (1993),  used  a  balaneed  scoreeard  to  assess  eross-funetional 
areas  within  an  organization  to  assure  local  optima  are  not  working  to  the  detriment  of 
other  funetional  areas.  Brown  later  expanded  this  work  through  a  strategie  model  to 
measure  the  balanced  seoreeard  as  eited  in  Graham  (1996).  Both  Kaplan  and  Brown 
viewed  performanee  metrics  as  the  key  suceess  faetors  aehieving  goals  and  objeetives  in¬ 
line  with  strategies.  Reeounting  Ammons  (1996)  study  of  the  performance  measures  of 
workload,  effieieney,  effectiveness,  and  productivity,  arguably  all  of  these  measures  are 
contained  within  the  factors  driving  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates. 
Though  this  projeet  does  not  result  in  a  eompleted  BCA,  the  metries  ehosen  exhibit  the 
ability  for  further  study  within  the  context  of  researeh  performed  by  Ammons  (1996)  and 
Kaplan  (1993). 
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In  terms  of  out-put,  cost,  and  quality,  the  benefit  metrics  chosen  for  this  project  all 
reflect  indicators  of  activity  levels  impacting  quality  metrics  tied  to  costs  (Ammons, 
1996).  Additionally,  they  serve  as  “normalization  factors”  to  facilitate  comparison 
between  activities  not  directly  comparable.  In  simpler  terms,  a  particular  sustainment 
strategy’s  attainment  of  low  NMCM  rates  draws  comparison  to  a  different  sustainment 
strategy  to  determine  areas  for  improvement  through  replication  or  adaptation  of 
successful  activities.  Each  activity  required  for  sustainment  and  evaluated  on  its  merit  to 
positively  impact  aircraft  availability  also  has  an  associated  cost.  Understanding  costs  as 
either  operating  costs  or  capital  costs  makes  possible  the  assignment  of  expenses  to 
activities  or  overhead  and  help  analyze  the  amount  of  productivity  an  activity  has.  Such 
an  understanding  is  possible  using  the  aforementioned  benefit  metrics.  Lastly,  quality 
measured  with  quantitative  metrics  gives  the  benefit  of  assessing  what  Ammons  (1996) 
states  as  difficult  to  ascertain  -  the  timeliness  and  thoroughness  of  repairs. 

Overview  of  Aircraft  Availability 

The  target  aircraft  availability  rate  of  72.9%  also  represents  a  standard  for  which 
the  individual  goals  of  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates  (along  with 
UPNR)  will  be  affected  by  (LIMS-EV  Data,  2013).  Target  aircraft  availability  has 
remained  unchanged  since  2009  (LIMS-EV,  2013). 

The  2009  BCA  used  the  benefit  metrics  of:  normalized  aircraft  availability  rate 
relative  to  fiscal  year  2008  targeted  goal,  on-time  delivery  rate  (original  and  revised), 
routine  depot  assistance  request  response  time,  indexed  not  mission  capable  supply  rate, 
component  product  quality  deficiency  report  rate,  customer  wait  time,  issue  effectiveness 


25 


rate,  and  stockage  effectiveness  rate  (BCA:  Executive  Summary,  2009).  While  the 
overall  construct  of  these  metrics  remained  intact  in  this  project,  there  were  several 
changes.  Grouped  together,  with  the  exception  of  on-time  delivery  rate,  the  remaining 
metrics  of  the  2009  BCA  can  all  be  measured  with  NMCM,  NMCS,  NMCB,  and  Depot 
Time  rates.  Furthermore,  through  measurement  of  these  four  metrics,  the  major  factors 
of  aircraft  sustainment  are  assessed  for  their  impact,  cost,  and  necessity.  In  other  words, 
over  a  long  period  of  time,  UPNR  rates  eventually  go  to  zero  for  the  specific  aircraft  as 
the  maintenance  status  is  specified  or  the  disposition  given.  Routine  depot  assistance 
request  response  time  is  a  measure  of  the  time  it  takes  a  depot  to  assist  with  a 
maintenance  disposition  but  there  is  anecdotal  evidence  to  suggest  this  process  is  not 
used  as  routinely  as  envisioned  due  in  part  to  the  hours  of  operation  at  depots  and  their 
subsequent  inability  to  furnish  an  answer  over  times  such  as  weekends  or  late  nights. 
From  this,  there  is  reason  to  believe  routine  depot  assistance  request  response  time  does 
not  completely  correlate  with  actual  needs  for  depot  assistance.  The  component  product 
quality  deficiency  report  rate  (PDQR)  was  not  included  in  this  project  because  PDQR 
rates  impact  NMCB  and  NMCS  and  consequently  represent  a  factor  measured  in  the 
2009  BCA’s  benefit  metrics  of  aircraft  availability  and  non-mission  capable  supply  rates. 
The  remaining  2009  BCA  benefit  metrics  of  customer  wait  time,  issue  effectiveness  rate 
and  stock  effectiveness  rate  are  also  components  of  indexed  non-mission  capable  supply 
rate.  Product  deficiency  rates  fail  to  demonstrate  mutual  exclusivity  since  they  impact 
NMCS,NMCB,  and  customer  wait  time. 
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Decision  Analysis 

Decision  analysis  is  a  tool  for  making  decisions  with  multiple,  competing  factors. 
It  represents  a  strategic  method  of  assessing  different  decision  criteria  to  inform  decision¬ 
makers  of  the  potential  impact  and  consequences  of  alternatives.  The  essential  element 
of  a  decision  is  the  presence  of  alternatives  (Kirkwood,  1997).  It  is  therefore 
instrumental  the  decision  criteria  be  mutually  exclusive  in  order  to  produce  alternatives. 
Value  hierarchies  represent  tiers  in  a  decision.  Since  there  is  no  single  metric  to  evaluate 
depot  performance,  aspects  of  the  depot  must  be  categorized  by  tier.  Such  a  hierarchy  is 
shown  in  Table  1:  Aircraft  Availability  Sub-Factors.  Aspects  of  aircraft  availability  are 
divided  by  tiers  based  on  their  applicable  parts.  Kirkwood  describes  this  organization  as 
a  value  hierarchy  (Kirkwood,  1997).  Aircraft  Sustainment  can  be  similarly  broken  into 
value  hierarchies  with  activities  broken  into  the  sub-components.  The  sub-components, 
or  tiers  within  the  value  hierarchy,  should  have  certain  desirable  properties.  These 
desirable  properties  represent  extensive  study  of  decision  theory  and  are  briefly  described 
in  the  ensuing  paragraphs.  Kirkwood  states  the  desirable  properties  for  value  hierarchies, 
those  used  in  the  2009  BCA  and  the  ones  proposed  by  this  project,  are:  Completeness, 
nonredundancy,  decomposability,  operability,  and  small  size  (Kirkwood,  1997). 

Completeness  is  a  measure  of  how  well  the  metrics  address  pertinent  concerns.  It 
is  necessary  to  know  only  how  well  an  alternative  performs  in  lowest-tier  evaluation 
considerations  assess  its  performance  with  respect  to  the  overall  evaluation  consideration 
(Kirkwood,  1997).  Kirkwood  further  states  if  all  important  evaluation  considerations  are 
not  included,  then  evaluating  alternatives  may  not  be  able  to  distinguish  those  of  different 
preferability  (Kirkwood,  1997).  In  simpler  terms,  if  the  metrics  do  not  include  all  the 
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important  measures,  the  decision  will  likely  be  unable  to  achieve  the  desired  objectives. 
Completeness  of  the  metrics  ensures  all  are  considered  and  have  a  distinctive  place  in  the 
decision  analysis. 

Nonredundancy  is  the  second  of  the  five  desirable  properties.  Nonredundancy 
ensures  no  two  evaluation  considerations  in  the  same  tier  overlap  (Kirkwood,  1997). 

This  is  also  an  essential  part  of  the  hierarchal  nature  of  the  chosen  metrics  -  the  mere 
presence  of  the  lower  tier  divides  up  the  metrics  above  it  with  more  detail  (Kirkwood, 
1997).  On  this  point,  the  2009  BCA  used  overlapping  tiers  of  metrics  to  determine 
benefits  for  further  analysis.  Nonredundancy  ensures  each  metric  receives  its  desired  and 
apportioned  weight  without  being  influenced  by  the  weights  of  superior  or  subordinate 
metrics.  Furthermore,  the  nonredundant  decision  criteria  fit  into  precisely  one  category. 

Decomposability,  often  referred  to  as  interdependence,  is  the  third  of  the  five 
desirable  properties.  Decomposable  decision  criteria  are  not  counted  in  more  than  one  of 
the  lower- tier  considerations  (Kirkwood,  1997).  That  is,  when  measuring  criteria  in  one 
tier,  it  is  inappropriate  to  then  again  measure  its  sub-components  in  another  tier.  A  lack 
of  decomposability  causes  skewed  analysis  when  determining  the  overall  preferability  of 
an  alternative  (Kirkwood,  1997). 

The  fourth  desirable  properties  is  operability.  Operability  relates  to  an 
understandable  and  interpretable  structure  -  both  intuitive  and  logical  in  its  structure  and 
organization  (Kirkwood,  1997).  Operability  is  often  a  subjective  term  -  some  who  lack 
the  specialization  or  understanding  of  the  criteria  may  not  be  able  to  readily  interpret  its 
inherent  importance.  Nonetheless,  the  decision  criteria  must  still  relate  to  the  model  used 
in  order  to  be  of  any  benefit  -  often  causing  those  with  less  experience  in  the  discipline  to 
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become  confused.  In  the  case  of  C-17A  sustainment  strategies,  the  operability  of  the 
decision  criteria,  or  metrics,  chosen  are  both  generalizable  across  other  aircraft  and 
transferable  to  those  strategies. 

Lastly,  small  size  is  a  desirable  property  of  decision  criteria.  A  relatively  small 
number  of  decision  criteria  can  be  communicated  more  easily  and  further  represent  where 
tradeoffs  can  be  made  between  decision  criteria  (Kirkwood,  1997).  The  small  size 
preference  also  ensures  only  the  most  important  and  relevant  information  is  considered 
(Kirkwood,  1997).  Certainly,  in  sustainment  strategies,  there  are  innumerable  details 
from  decisions  on  economic  order  quantities  of  bench  stock  items  to  performance 
evaluation  criteria  for  first,  second,  and  third  tier  supply  chain  partners.  However,  the 
ultimate  reflection  of  those  decisions  can  be  captured  in  an  over-arching  metric.  The 
authors  of  this  project  argue  those  over-arching  metrics  are  NMCS,  NMCM,  NMCB,  and 
Depot  Time  rates  for  the  entire  fleet. 

The  use  of  decision  analysis  as  a  tool  for  further  informing  strategic  decisions 
based  on  their  ability  to  achieve  desirable  objectives  is  greatly  enhanced  though  careful 
selection  of  decision  criteria.  According  to  Kirkwood,  the  desirable  properties  of  those 
decision  criteria  are  completeness,  nonredundancy,  decomposability,  operability,  and 
small  size.  This  belief  of  this  project  is  that  the  metrics  chosen  reflect  these  properties 
and  create  opportunities  within  this  trade  space  for  the  consideration  of  alternatives  with 
respect  to  the  sustainment  strategies. 
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III.  Methodology 


Chapter  Overview 

The  purpose  of  this  section  is  to  detail  the  approach  taken  to  assess  the  impact  of 
different  sources  of  repair  on  the  sub-factors  of  aircraft  availability.  This  project  used  a 
similar  methodology  as  the  2009  BCA  with  respect  to  developing  a  decision  analysis- 
based  model  to  evaluate  performance  metrics  for  C-17A  sustainment.  The  metrics 
chosen  were  the  result  of  an  in-depth  review  of  the  2009  BCA,  discussion  with  the  C-17A 
Program  Office  at  AFLCMC,  senior  leaders  in  the  AFMC’s  AFSC,  subject  matter  experts 
in  depot  maintenance,  graduated  aircraft  maintenance  squadron  commanders,  and 
doctorate  level  Operations  Research  faculty.  Refined  aircraft  data  was  used  in  the  model 
created  to  evaluate  the  impact  different  sources  of  repair,  corresponding  to  different 
sustainment  strategies,  had  on  the  sub-factors  of  aircraft  availability. 

Aircraft  Selection  and  Sustainment  Strategies 

The  aircraft  chosen  for  this  project  were  representative  of  the  different 
sustainment  strategies  currently  in  use  throughout  the  USAF  with  respect  to  tanker, 
airlift,  and  reconnaissance  and  surveillance  aircraft.  Variations  of  certain  sustainment 
strategies  were  by  either  separating  or  combining  aircraft  models  indicative  of  that 
sustainment  strategy. 

Of  important  note,  two  facts  bear  mentioning.  First,  the  sustainment  strategy  and 
aircraft  sustainment  activities  cannot  be  separated  from  each  other.  In  other  words,  it 
would  be  difficult  to  know  if  the  aircraft  and  its  inherent  maintainability  are  responsible 
for  the  aircraft  availability  sub-factor  values  or,  whether  those  values  are  tied  to  the 
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sustainment  strategy  itself.  The  projeet  did  not  use  normalized  data  for  this  reason  as 
well.  Though  the  2009  BCA  normalized  data  relative  to  a  fixed  standard,  a  different 
methodology  was  used  for  this  projeet.  Seeondly,  the  only  sustainment  strategies 
assessed  were  the  current  ones  in  use.  The  2009  BCA  created  sustainment  options,  then 
used  extrapolated  data  to  predict  performance  over  a  30-year  period.  The  limited  options 
of  sustainment  strategies  currently  in  use  were  defendable  through  data  and  therefore 
represented  potential  sustainment  strategies  to  transition  the  C-17A  to.  Accompanying 
each  aircraft  is  a  short  summary  of  the  sustainment  strategy  in  use. 

E-8C 

The  Northrop  Grumman  E-8C  Joint  Surveillance  Target  Attack  Radar  System  (J- 
STARS)  was  initially  produced  in  1997  (E-8C,  2012).  The  E-8C  is  based  off  a  modified 
Boeing  707  airframe  and  has  a  distinctive  27  foot  long  radome  along  its  underside 
housing  a  phased-array  radar  (E-8C,  2012).  The  E-8C  conducts  battle  management  and 
command  and  control  missions  in  support  of  targeting  operations.  The  E-8C  was 
included  in  this  project  because  it  represents  a  performance  based  logistics  (PBE) 
sustainment  strategy  in  addition  to  organic  depot  activities.  Northrop  Grumman  sustains 
the  engines  in  conjunction  with  a  prime  contractor,  Pratt  &  Whitney.  Most  recently,  the 
E-8C  has  undergone  a  re-engining  process  to  upgrade  and  its  four  turbofan  engines.  The 
majority  of  the  re-engining  work  was  conducted  at  Northrop  Grumman’s  Take  Charles, 
Eouisiana  Depot.  The  remaining  sustainment  activities  occur  at  the  WR-ALC  (Northrop, 
2013). 


31 


KC-10 


The  KC-10  Extender  is  the  largest  refueling  aireraft  in  the  world.  In  addition  to 
its  ability  to  refuel  other  aircraft,  it  has  an  extensive  cargo  mission.  Based  on  the 
McDonnell  Douglas  DC-10  airliner,  the  KC-10  can  carry  170,000  pounds  of  cargo  (KC- 
10,  201 1).  The  KC-10  was  included  in  this  project  because  it  represents  a  larger  scale 
PBL  sustainment  strategy.  Currently,  Northrop  Grumman  is  responsible  for:  line 
replaceable  units,  test  and  support  equipment,  consumable  parts,  bench  stock  items,  and 
flight  line  support  and  maintenance  equipment  (KC-10  Contractor,  2012).  The  remaining 
items  of  sustainment  are  primarily  sourced  at  the  WR-ALC.  The  E-8C  represents  a 
unique  blend  of  PBE  activities  with  certain  functions  performed  at  organic  depots  in 
order  to  maintain  those  capabilities  within  the  government. 

C-5A  &  B 

The  C-5A  and  B  models  were  built  in  1970  and  1986,  respectively,  by  the 
Eockheed-Georgia  Company  (C-5A/B,  2012).  The  C-5  A  and  B  are  strategic  airlift 
aircraft  with  a  cargo  capacity  of  270,000  pounds.  The  C-5  A  and  B  models  were 
included  in  this  project  because  they  represent  an  organic  sustainment  strategy  with  AEC- 
directed  sustainment  activities  (WR-AEC,  2011).  The  WR-ALC  uses  the  same  organic 
depots  for  upgrades  and  the  modification  program  that  takes  a  C-5A  or  B  and  transitions 
it  into  a  C-5M  through  a  two-phased  process.  Consequently,  there  are  fewer  C-5A  and  B 
model  aircraft  as  there  are  increases  to  the  C-5M  fleet. 
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C-130E  &  H 


The  C-130E  and  H  models  were  built  in  1962  and  1974,  respeetively,  by  the 
Loekheed-Martin  Aeronautics  Company  (C-130,  2011).  The  C-130  is  a  tactical  airlifter 
with  a  cargo  capacity  of  36,500  pounds.  The  C-130E  and  H  models  were  included  in  this 
project  because  they  represent  another  aircraft  using  an  organic  sustainment  strategy  with 
different  results  from  the  organic  sustainment  strategy  for  the  C-5.  The  WR-ALC 
coordinates  or  executes  the  coordination  of  all  facets  of  C-130E  and  H  sustainment.  This 
program  has  existed  with  minor  changes  throughout  the  evaluation  period  for  this  project. 
The  WR-ALC  also  conducts  programmed  and  unscheduled  depot  level  maintenance, 
coordinates  contract  and  depot  field  teams,  and  organic  and  contractor  support  (330* 
ASG,  2008). 

KC-135R 

The  KC-135R  was  initially  delivered  in  1957  by  Boeing  (KC-135R,  2011).  Based 
on  the  Boeing  707,  the  aircraft  is  the  cornerstone  of  the  refueling  fleet  and  can  carry 
83,000  pounds  of  cargo  as  well  (KC-135R,  2011).  The  KC-135R  was  included  in  this 
project  because  it  represents  a  contractor-based  sustainment  strategy.  Recently 
transitioned  to  Boeing  in  conjunction  with  several  other  contractors  in  1999,  the  KC- 
135R  represents  a  significant  sustainment  strategy  success  as  the  average  fleet  age  is  57 
years  and  predicted  to  surpass  80  years  before  it  is  retired  (KC-135R,  2011). 

Benefit  Model  Description 

The  benefit  model  used  in  this  project  used  the  2009  BCA  as  its  basis.  The 
weights  represented  prioritized  values  as  previously  described.  The  rate  columns  were 
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populated  with  the  average  values  for  the  speeifie  metrie  assessed.  Averages  were  taken 
from  the  speeified  duration  and  then  plaeed  into  the  cell  corresponding  to  the  aircraft 
within  the  sustainment  type  noted  above  the  model.  Individual  averages  were  multiplied 
by  the  weighted  value  assigned  to  the  particular  benefit 

The  purpose  of  the  benefit  model  is  to  assess  the  impact  of  different  sources  of 
repair  on  the  sub-factors  of  aircraft  availability.  Historical  data  from  several  major 
weapons  systems  was  collected  from  Logistics,  Installations,  and  Mission  Support  - 
Enterprise  View  (LIMS-EV)  and  the  Air  Eorce  Total  Ownership  Cost  (AETOC) 
databases.  Historical  data  for  the  C-17A,  C-5A,  B,  and  M  models,  C-130E,  H,  and  J 
models,  KC-135  R,  KC-10,  and  E-8A  was  collected  from  EY1994  to  EY2013.  Data  over 
this  period  of  time  was  analyzed  for  anomalies  that  may  skew  analysis  and  vetted  for  the 
impact  of  program  transitions,  maturity  of  the  reliability  curve,  and  number  of  aircraft  in 
the  fleet.  Data  previous  to  EY1994  was  considered  either  too  dated  or  unrepresentative 
of  the  current  practices  in  the  view  of  the  authors  of  this  project. 

Several  findings  from  this  study  led  to  the  creation  of  business  rules  to  guide  the 
refinement  of  data,  the  exclusion  of  certain  aircraft  altogether,  or  the  exclusion  of  certain 
years  of  aircraft  data.  Data  for  the  sustainment  strategies  for  the  C-5M  and  C-130J 
aircraft  were  completely  excluded  for  this  project  because  of  the  significant  amount  of 
learning  still  in  progress.  Eearning,  in  terms  of  maintainers,  logistics,  supply  chain 
management  policies  and  technical  data  development  will  eventually  result  in  lower 
NMCS,  NMCM,  NMCB,  and  Depot  Time  rates.  However,  presently  the  sustainment 
strategy  data  these  two  programs  are  not  yet  representative  of  the  program  and  therefore 
were  not  included.  Data  for  the  first  five  years  after  initial  production  and  the  first  three 
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years  after  a  transition  were  excluded.  Initial  production  exclusion  accounted  for  the 
growth  in  the  reliability  of  the  aircraft  and  improvements  in  maintenance  and  logistics 
through  learning.  Data  for  the  first  three  years  after  a  transition  to  a  new  source  of 
sustainment  was  also  excluded  to  account  for  the  transition  costs.  Lastly,  the  final  three 
years  of  an  aircraft’s  sustainment  program  data  was  excluded.  This  final  three  years  of  an 
aircraft  program,  such  as  the  C-130E,  demonstrates  low  fleet  numbers  and  high  amounts 
of  cannibalization  that  are  not  indicative  of  the  sustainment  strategy  previously  used. 

Data  for  C-17A,  C-5A,  and  B,  KC-10,  KC-135R,  E-8C,  and  C-130  E,  and  H 
models  were  imported  into  MS  Excel®  arranged  by  fiscal  year.  Columns  were  added  for 
NMCS,  NMCM,  NMCB,  and  Depot  Time  rates.  All  columns  for  NMCS,  NMCM, 
NMCB,  and  Depot  Time  rate  data  were  then  averaged  as  a  percentage  rate  for  the  entire 
duration  of  fiscal  years  assessed  (see  Appendix  B,  Table  2  for  individual  metric 
formulas). 

The  development  of  the  weight  for  NMCS  was  derived  from  the  2009  BCA  by 
totaling  the  percentage  of  supply-related  functions,  discussing  appropriate  weighted 
values  with  the  C-17A  program  office  at  WR-ALC,  graduated  maintenance  squadron 
commanders,  and  senior  leaders  at  the  Air  Eorce  Sustainment  Center  (AESC).  Each  of 
these  aircraft  availability  sub-factors  was  multiplied  by  a  specified  weight  to  show  the 
average  weighted  impact  for  the  sustainment  program.  All  the  metrics  related  to  a 
derivative  of  the  supply  function  were  totaled  for  their  relative  importance  to  the  whole 
model.  The  summation  of  all  the  derivatives  of  the  supply  function  from  the  2009  BCA 
was  42.2%.  This  approximate  percentage  was  reflected  in  the  model  this  project 
proposes  resulting  in  a  value  of  40%  assigned  to  the  derivative  of  the  supply  function. 
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The  proportionate  value  was  maintained  relative  to  the  other  sub-factors  of  aircraft 
availability  used  in  this  model.  The  weight  for  NMCM  was  evaluated  to  75%  as 
important  as  NMCS.  The  resulting  weight  for  NMCM  was  30%.  The  weight  for  NMCB 
was  not  addressed  in  the  2009  BCA.  Consequently,  the  weighted  value  of  this  metric 
was  subject  to  discussions  within  the  academic  environment,  between  several 
maintenance  experts  having  successfully  completed  command  of  an  aircraft  maintenance 
squadron,  senior  leadership  from  the  Air  Force  Sustainment  Center,  and  lastly,  from  the 
C-17A  Program  Management  Office  at  the  AFLCMC.  The  resulting  weight  was 
established  at  10%.  The  weight  for  Depot  Time  was  also  not  addressed  in  the  2009  BCA 
and  underwent  the  same  assessment  as  the  NMCB  weight.  The  resulting  weight  for 
Depot  Time  rate  was  20%.  Further  development  of  weights  should  be  informed  by  senior 
leaders  based  on  strategic  priorities. 

Sensitivity  analysis  was  then  performed  on  the  model.  The  results  of  the 
sensitivity  analysis  are  found  in  Figure  2  below.  The  sensitivity  analysis  performed 
sought  to  keep  the  same  proportion  of  weighted  values  intact  as  they  were  either  added  or 
subtracted  to  the  weights.  Specifically,  for  every  4-point  increase  in  the  value  of  NMCS, 
there  was  a  corresponding  decrease  in  the  values  of  NMCM,  NMCB,  and  Depot  Time 
rates  by  3  points,  1  point  and  2  points,  respectively.  This  process  was  repeated  until  a 
change  occurred  in  the  sustainment  option  with  the  lowest  calculated  value.  After 
excluding  the  C-130J  program  because  its  Depot  has  only  been  in  operation  since  May, 
2011,  the  only  other  programs  possessing  lower  values  for  the  sub-factors  measured  were 
the  KC-10  (NMCB  rate  and  Depot  Time  rate)  and  the  C-130E  &  H  models  (Depot  Time 
rate  only). 


36 


IV.  Analysis  and  Results 


Chapter  Overview 

The  results  of  analyzing  NMCS,  NMCM,  NMCB,  and  Depot  Time  rates  are 
shown  in  Figures  3  through  6.  These  figures  represent  charts  of  aircraft-specific  rates 
from  data  collected  over  the  described  period.  Figures  7  through  1 1  show  the  individual 
aircraft  raw  data  used  to  complete  this  analysis  with  shaded  regions  representing  data 
excluded  from  analysis. 

This  project  found  the  best  strategy,  according  to  the  benefit  model  only,  to 
reduce  the  rates  of  NMCS,  NMCM,  NMCB,  and  a  Depot  Time  rate  was  the  current 
sustainment  program  for  Option  1  in  Figure  2,  the  C-17A.  It  represents  the  best 
performing  sustainment  strategy  in  terms  of  benefits.  However,  cost  analysis,  risk 
assessment,  or  incentive  structure  analysis  has  yet  to  be  accomplished  to  accompany 
these  results.  Until  the  completion  of  research  to  fully  inform  the  next  BCA,  the  resultant 
best-value  strategy  is  not  known. 
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Table  1:  Benefit  Metrics 


PSM/PSI 

PBL 

PBL 

Organic 

Organic 

Contractor 

Benefits  Metric 

Weights 

Option  1 

PSM/PSI 

C-17A 

Option  2A 

PBL 

E-8C 

Option  2B 

PBL 

KC-10 

Option  3A 

Organic 

C-5A,B 

Option  3B 

Organic 
C-130  E,H 

Option  4A 

Contractor 

KC-135R 

Percent  Non-Mission  Capable 
Maintenance  Rate  (NMCM) 

Rate 

0.0906 

0.0971 

0.1001 

0.1824 

0.1160 

0.1096 

30.00% 

Weighted 

Result 

0.0272 

0.0291 

0.0300 

0.0547 

0.0348 

0.0329 

Percent  Non-Mission  Capable 
Supply  Rate  (NMCS) 

Rate 

0.0209 

0.0368 

0.0254 

0.0508 

0.0286 

0.0294 

40.00% 

Weighted 

Result 

0.0083 

0.0147 

0.0102 

0.0203 

0.0114 

0.0118 

Percent  Non-Mission  Capable 

Both  Rate  (MNCB) 

Rate 

0.0150 

0.0171 

0.0137 

0.0744 

0.0670 

0.0342 

10.00% 

Weighted 

Result 

0.0015 

0.0017 

0.0014 

0.0074 

0.0067 

0.0034 

Rate 

0.1433 

0.2689 

0.1410 

0.1738 

0.1054 

0.1488 

Depot  Time  Rate 

20.00% 

Weighted 

Result 

0.0287 

0.0538 

0.0282 

0.0348 

0.0211 

0.0298 

Sum  of  individual  metrics  weighted  results 

0.0657 

0.0993 

0.0697 

0.1172 

0.0740 

0.0778 

Rank  Order 

1 

5 

2 

6 

3 

4 

FY  Begin 

FY98 

FY01 

FY94 

FY94 

FY02 

FY99 

FYEnd 

FY13 

FY13 

FY13 

FY13 

FY13 

FY13 

Fleet  Size  FY13 

217 

17 

59 

31(A)/31(B) 

6(E)/265(H) 

362 

Average  Fleet  Size 

135 

17 

59 

56 

221 

356 

Years  Since  Milestone  C 

20 

15 

32 

43(A)/27(B) 

51(E)/39(H) 

57 

Results  of  Sensitivity  Analysis 

Two  key  principles  were  applied  to  the  sensitivity  analysis.  First,  there  were  only 
two  aircraft  sustainment  programs  with  benefit  metric  values  less  than  Option  1 ,  the  C- 
17A.  Those  programs  with  lower  benefit  metric  values  were  Options  2B,  the  KC-10  and 
Option  3B,  the  C-130  E  and  H  model  sustainment  programs.  The  KC-10  NMCB  and 
Depot  Time  rates  were  lower  than  the  C-17A  and  the  C-130  E  and  H  models  had  lower 
Depot  Time  rates  than  the  C-17A  as  well.  No  other  programs  expressed  lower  values  of 
the  four  metrics  assessed.  Consequently,  no  change  in  the  weighted  values  would  cause 
those  programs  with  higher  values  to  become  lower  than  the  C-17A.  From  this  finding, 
sensitivity  analysis  focused  on  those  two  programs,  excluding  all  others,  and  sought  to 
determine  how  much  the  NMCB  and  Depot  Time  weights  needed  to  be  changed  in  order 
to  produce  a  different  result  in  the  model. 
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The  summation  of  weights  totaled  100%  and  corresponding  changes  to  the 
weights  associated  with  the  sensitivity  analysis  did  not  exceed  one  hundred  percent. 
Changes  were  taken  incrementally  to  affect  the  overall  results  of  the  analysis.  By 
changing  the  weighted  values  associated  with  the  benefit  metrics,  certain  values  were 
noted  which  would  change  the  results  and  conclusions  of  this  analysis.  The  initial 
weighted  values  were  reduced  by  three  percent  for  NMCS  rate  and  four  percent  for 
NMCM  rate,  while  the  weighted  values  were  increased  by  three  percent  and  four  percent 
for  NMCB  rate  and  Depot  Time  rate,  respectively.  The  values,  as  assigned  from  the 
study,  their  corresponding  rank  orders  in  terms  of  lowest  value,  and  the  values  needed  to 
change  these  recommendations  are  found  in  the  figure  below  (Figure  X-X).  Under  no 
weighted  combination  did  the  sustainment  program  embodied  by  the  C-130E  and  H 
models  outperform  the  C-17A  overall.  Those  sustainment  programs  without  any  values 
less  than  the  C-17A  sustainment  program  were  removed  from  the  sensitivity  analysis  and 
are  represented  by  “grayed-out”  sections  with  bold  “X’s”.  Additionally,  the  highlighted 
cells  were  the  only  cells  with  values  less  than  those  observed  by  the  C-17A  sustainment 
program.  The  sensitivity  analysis  sought  to  determine  what  the  overall  weights  had  to  be 
in  order  for  these  sustainment  strategies  to  outcompete  the  current  C-17A  sustainment 
program. 
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Table  2:  Sensitivity  Analysis 


PSM/PSI 

\  1 

PBL 

^  Organic  i 

Organic 

\Contractor, 

Benefits  Metric 

Weights 

Option  1 

PSM/PSI 

C-17A 

toption  Ifii 

\  / 

\e-8C  / 

Option  2B 

PBL 

KC-10 

\Option  3A/ 

\  Organic  / 
\c-5A,b/ 

Option  3B 

Organic 

C-130E,H 

option  Afi/ 

®ontract(x 

l\c-135R 

Percent  Non-Mission  Capable 
Maintenance  Rate  (NMCM) 

9.00% 

Rate 

0.0906 

Vq97J 

0.1001 

Yi82y 

0.1160 

Vl099 

Weighted 

Result 

0.0082 

owop 

0.0090 

noip 

0.0104 

o)p099 

Percent  Non-Mission  Capable 
Supply  Rate  (NMCS) 

12.00% 

Rate 

0.0209 

0.^68 

0.0254 

o.oi/os 

0.0286 

0.0294 

Weighted 

Result 

0.0025 

0.G^4 

0.0030 

O-iMI 

0.0034 

0.^5 

Percent  Non-Mission  Capable 
Both  Rate  (MNCB) 

31.00% 

Rate 

0.0150 

0.0137 

(f.074A 

0.0670 

Weighted 

Result 

0.0046 

/.005\ 

0.0042 

p.023\ 

0.0208 

/.oio\ 

Depot  Time  Rate 

48.00% 

Rate 

0.1433 

/o.2689\ 

0.1410 

/o.1738\ 

0.1054 

/o.1488\ 

Weighted 

Result 

0.0688 

/  0.1291  \ 

0.0677 

/  0.0834  \ 

0.0506 

/  0.0714  \ 

Sum  of  individual  metrics  weighted  resultsi  0.0841  |  0.1475  |  0.0839  |  0.1290  |  0.0852  |  0.0954  | 


FY  Begin 

FY98 

FY01 

FY94 

FY94 

FY02 

FY99 

FYEnd 

FY13 

FY13 

FY13 

FY13 

FY13 

FY13 

Fleet  Size  FY13 

217 

17 

59 

31(A)/31(B) 

6(E)/264(H) 

362 

Average  Fleet  Size 

135 

17 

59 

56 

221 

356 

Years  Since  Milestone  C 

20 

15 

32 

43(A)/27(B) 

51(E)/39(H) 

57 

The  figure  shows  one  possible  change  required  in  the  values  for  the  weighted 
benefit  metrics  required  in  order  to  achieve  alternate  outcomes.  The  sensitivity  analysis 
shows  significant  changes  in  the  values  of  metrics  in  order  to  drive  changes  in  the  model. 
This  dramatic  change  in  weighted  values  demonstrates  the  dominance  of  the  current 
sustainment  strategy’s  performance  over  those  in  the  comparison  across  all  aspects  of 
sub-factors. 

Investigative  Questions  Answered 

The  research  questions  posed  in  this  study  sought  to  provide  both  the  justification 
of  the  study  and  segue  into  the  creation  of  alternate  sustainment  performance  metrics  to 
incorporate  into  future  business  cases  analyses.  The  answers  to  the  questions  found  in 
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the  Section  I;  Research  Questions,  drove  much  of  the  analysis,  framing,  findings,  and 
ultimately  the  results  and  conclusions  of  this  study. 

The  first  investigative  question  centered  on  the  mutual  exclusivity  of  the  benefit 
metrics  used  in  the  2009  BCA.  In  order  for  a  criterion  to  be  mutually  exclusive  it  must 
not  be  assigned  to  more  than  one  category.  A  criterion  is  assigned  to  more  than  one 
category  effects  on  each  category  thus  reducing  the  margin  of  difference  between  criteria. 
When  a  decision  criterion  exhibits  mutually  inclusive  traits,  the  causal  impact  cannot  be 
determined.  A  single  trait  shared  among  decision  criteria,  in  the  case  of  the  2009  BCA, 
the  benefits  metrics,  cannot  be  separated  from  other  decision  criterion  sharing  that  same 
trait.  Despite  different  terminology  and  the  fit  with  what  Ammons  (1996),  Kaplan 
(1993),  and  Brown  (1996)  deem  as  necessary  attributes  for  evaluating  performance 
measures,  the  2009  BCA  benefits  metrics  are  clearly  mutually  inclusive.  Simply  put, 
from  the  metrics  chosen,  there  is  no  reason  to  believe  a  broken  part  off  the  shelf  at  the 
depot  will  not  require  a  new  part  to  be  ordered,  increasing  customer  wait  time,  decreasing 
stockage  effectiveness  rate  while  driving  up  the  overall  time  the  weapons  systems  in  non¬ 
mission  capable  for  supply  and  degrading  the  aircraft’s  availability.  The  mutually 
inclusive  trait  across  all  these  decision  criteria  was  the  single  part  needed  to  make  the 
aircraft  operational.  One  measure  for  the  supply  function  would  be  sufficient  to  capture 
this  occurrence  while  preventing  a  stockout  or  deficient  part  from  impacting  several  other 
decision  criteria. 

From  the  2009  BCA,  the  method  used  to  calculate  Normalized  Aircraft 
Availability  Rate  Relative  to  the  Aircraft  Standard  represented  the  percentage  of  aircraft 
not  in  a  depot  status  or  in  a  non-mission  capable  status.  The  formula  for  determining 


41 


aircraft  availability  is:  Aircraft  Availability  =  100%  -  NMCS  rate  -  NMCM  rate  - 
NMCB  rate  -  Depot  Time  -  UPRN  rate.  Using  aireraft  availability  in  addition  to  its  sub- 
faetor  as  another  metrie  result  in  an  interaetion  between  these  deeision  eriteria  that  skews 
analysis.  The  same  holds  true  for  Customer  Wait  Time.  As  NMCS  rates  deerease, 
Customer  Wait  Time  deereases  based  on  the  availability  of  parts  or  supplies  -  the  very 
faetor  influeneing  NMCS  rates.  Additionally,  NMCS  rates  exhibit  interaetions  with  Issue 
Effectiveness  Rates  sinee  the  eommon  faetor  between  the  two  is  number  of  baekordered 
parts.  As  NMCS  Rates  deerease  based  on  fewer  baekorders,  Issue  Effeetiveness  Rates 
also  deerease  based  on  fewer  baekorders.  Stoek  Effectiveness  Rates  are  prone  to  same 
sub-faetor  interferenee  as  NMCS  Rates  due  to  the  shared  baekorder  metric. 

Of  the  metrics  used  in  the  2009  BCA,  there  are  only  three  faetors  whieh  do  not 
exhibit  some  degree  of  sub-faetor  interaetion:  ON-Time  Delivery  Rate  (Original  and 
Revised),  Routine  Depot  Assistanee  Request  Response  Time,  and  Component  Product 
Quality  Deficiency  Report  Rate.  The  remaining  five  metries  used  in  the  2009  BCA  use 
mutually  inelusive  metries  with  varying  impacts  throughout  each  of  the  metrics  and 
therefore,  throughout  the  entire  analysis  and  should  not  be  used  together. 

The  second  research  question  focused  on  the  2009  BCA  weights  and  their  souree 
for  determination.  The  2009  BCA  weights  were  determined  using  a  survey  of  air 
logisties  eenter  (ALC)  subject  matter  experts.  Strategic  stakeholders  assigned  weights 
indieative  of  their  pereeived  priority  of  the  individual  benefit  metrie.  The  rigor  and  the 
aeademie  integrity  of  the  proeess  for  assigning  priority  and  weights  are  both  appropriate 
and  eorreetly  used.  This  represents  a  eonsiderable  strength  in  the  2009  BCA 
methodology  in  its  ability  to  survey  individual  stakeholders  and  subject  matter  experts. 
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The  third  research  question  focused  on  the  correctness  of  the  weights  used  in  the 
2009  BCA.  From  discussion  with  subject  matter  experts,  there  is  reason  to  believe  the 
intentionally  higher  priority  given  to  the  supply  function  is  both  appropriate  and 
corresponds  to  intuitive  logic  regarding  sustainment.  If  parts  are  available  for  trained 
maintainers  to  use,  aircraft  will  likely  spend  less,  if  any,  time  in  a  NMCS  status  and  only 
be  recorded  in  NMCM  rates  as  the  part  is  appropriately  used.  Should  the  supply  function 
be  impeded,  aircraft  will  tend  to  spend  more  time  in  NMCS  status  followed  by  a  period 
of  time  in  an  NMCM  status  while  being  repaired. 

The  fourth  research  question  posed  sought  to  determine  which  metrics  could  best 
assess  the  impact  on  aircraft  availability  given  changes  in  the  source  of  repair.  To 
determine  which  metrics  best  assess  sustainment  source  performance,  the  measures  of 
sustainment  broadly  capturing  those  activities  pertinent  to  aircraft  longevity  and 
availability  should  be  used.  As  evidenced  with  such  initiatives  as  the  C-130  High 
Velocity  Maintenance  (HMV),  there  have  been  decreases  observed  in  NMCS,  NMCB, 
and  Depot  Time  rates  when  efforts  to  decrease  these  very  metrics  become  part  of  the 
sustainment  strategy  (Scully,  2009).  Sustainment  is  directly  tied  to  aircraft  availability. 
Long-term  solutions  to  keep  aircraft  flying  are  the  cornerstone  to  sustainment  strategies. 
Aircraft  availability  focuses  on  reducing  the  effects  of  five  factors,  or  decision  criteria, 
seeking  to  find  a  sustainable  balance  between  them.  The  five  factors  related  to  aircraft 
availability  are:  NMCS,  NMCM,  NMCB,  UPNR,  and  Depot  Time  Rates.  This  project 
chose  to  use  the  major  sub-factors  of  aircraft  availability  to  assess  sustainment 
performance.  The  only  factor  used  to  calculate  aircraft  availability  not  incorporated  into 
the  model  was  UPNR.  Unit  possessed  not  reported  was  excluded  from  the  model  based 
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on  the  short-lived  duration  of  the  status  and  the  high  potential  for  this  metric  to  exhibit 
sub-factor  interference  with  other  metrics  used.  An  aircraft  awaiting  a  decision  on  the 
specific  part  number  required  for  a  fix  or  an  aircraft  with  a  new  problem,  previously 
unseen  or  unanticipated  by  the  maintenance  function  will  enter  into  the  UPNR  status 
while  the  specific  part  or  maintenance  action  (or  both)  is  determined.  Aircraft  in  UPNR 
status  represent  a  method  to  still  count  that  specific  aircraft  against  aircraft  availability 
while  NMCS,  NMCM,  NMCB,  or  Depot  Time  status  is  assigned.  The  model  and  data 
exclusion  also  sought  to  minimize  the  impact  of  this  specific  status  insofar  as  eliminating 
the  time  until  the  weapon  system  matured  along  the  reliability  growth  curve  where  UPNR 
rates  are  observed  to  decrease  with  the  maturation  of  the  weapon  system  and  improved 
training  and  diagnosis  practices. 

The  final  research  question  sought  to  determine  the  appropriate  sources  of  data  for 
sustainment  strategies.  Once  the  metrics  were  chosen  and  validated  for  mutual 
exclusivity,  data  sources  were  explored  to  ensure  standardization  and  compatibility  with 
the  model.  The  LIMS-EV  database  was  used  to  retrieve  validated  data  on  the  entire  fleet 
of  Air  Force  aircraft  starting  from  fiscal  year  1991 .  Previous  years’  data  was  both  not 
available  and  also  not  reflective  of  current  sustainment  practices  in  use  today. 

Summary 

The  research  questions  represented  a  framework  for  this  study  to  guide  research 
and  the  development  of  the  model.  From  the  DoDIG  report,  there  was  a  specific  need  for 
a  comprehensive  BCA.  The  2009  BCA,  though  representative  of  progress  toward  this 
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option,  ultimately  fell  short  of  achieving  its  directed  purpose  of  analyzing  all  alternatives 
to  determine  the  best-value  sustainment  option. 


V,  Conclusions  and  Recommendations 


Discussion 

The  purpose  of  this  project  was  two-fold:  to  develop  mutually  exclusive  BCA 
benefit  metrics  to  assess  sustainment  strategies  and  to  develop  a  model  to  use  within  a 
decision  analysis  framework  to  inform  future  BCAs.  This  research  represents  the  first 
phase  in  the  construction  of  the  next  BCA  as  corresponding  research  will  work  to  provide 
risk  assessments  and  associated  costs  to  complete  such  analysis.  The  intent  of  this 
research  is  to  begin  to  develop  a  defendable  methodology  to  conduct  the  BCA  for  the  C- 
17A  by  providing  a  potential  means  for  capturing  performance  benefits  from  one  aircraft 
type  and  transferring  them  to  the  C-17A  by  adopting  or  adapting  those  policies  and 
structures  responsible  for  the  desired  performance.  Furthermore,  one  potential  for  future 
research  is  the  creation  of  real-time  data  display  in  the  form  of  a  Continuous  BCA  to 
inform  budgetary  decisions  for  all  aircraft. 

Conclusions  of  Research 

In  conclusion,  this  research  found  four  mutually  exclusive  performance  metrics 
for  consideration  in  the  next  business  case  analysis.  Those  metrics:  NMCS.  NMCM, 
NMCB,  and  Depot  Time  rates  were  assigned  weights  based  on  the  2009  parent 
document,  academic  research  and  advisement,  C-17A  Program  Management  Office 
analysts  as  well  as  input  from  the  Air  Force  Sustainment  Center  senior  staff.  The  model 
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generated  from  this  researeh  found  the  eurrent  sustainment  strategy  as  the  most  benefieial 
in  terms  of  performance  across  the  metrics  measured. 

Significance  of  Research 

The  objective  of  this  research  is  to  improve  upon  a  robust  2009  BCA.  Mutually 
exclusive  metrics  have  the  best  chance  of  delineating  between  different  options  where 
valuable  decisions  within  the  trade  space  can  be  decided.  When  used  in  a  decision 
analysis  framework,  the  absence  of  sub-factor  interference  allow  for  the  evaluation  of  the 
merits  of  the  independent  metrics. 

Though  the  results  of  this  project  are  applicable  only  to  the  C-17A,  there  is 
potential  for  generalizability  to  other  airframes.  The  2009  BCA  aided  in  informing  the 
weights  of  the  benefit  model  used.  Additional  research,  discussion,  and  surveys  may  be 
needed  to  provide  more  input  to  the  specific  weights  used  in  the  model.  However,  from 
the  sensitivity  analysis,  there  is  reason  to  believe  significant  changes  in  the  weights  are 
unlikely  to  produce  different  results  should  those  weights  be  changed  as  they  relate  to 
different  aircraft  and  models.  It  should  also  be  noted  the  transition  of  the  C-17A  to  a 
different  sustainment  program  may  negatively  impact  the  sub-factors  of  aircraft 
availability  for  a  period  of  time  observed  by  this  project  to  be  up  to  three  years  if  trends 
follow  previous  transitions  in  sustainment  strategy. 

Recommendations  for  Future  Work 

While  this  research  represents  only  the  performance  benefits  of  certain 
sustainment  structure,  future  research  should  include  aspects  of  sustainment  that  do  not 
involve  determining  the  appropriate  mix  of  government  and  contractor  activities.  Future 
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research  should  also  include  analysis  of  incentive  programs  and  their  potential  to  affect 
cost  savings  while  targeting  specific  performance  standards.  Of  note,  exceeding 
performance  standards  may  not  represent  the  best-value  option  for  C-17A  sustainment 
with  regard  to  cost  -  the  excess  expenditures  of  resources  to  exceed  standards  may  also 
represent  areas  for  future  savings  while  continuing  to  maintain  readiness  and  aircraft 
availability.  Moving  forward  from  this  research,  certain  incentive  strategies  and 
structures  should  be  discussed  in  order  to  improve  on  the  already  remarkable 
performance  record  of  C-17A  sustainment  while  realizing  the  imperative  for  cost  savings. 

The  next  step  is  to  re-assess  the  risk  metrics  associated  with  C-17A  sustainment 
using  mutually  exclusive  decision  factors  then  complete  the  cost  analysis  portion  to 
complete  the  next  BCA.  Cost  analysis  should  provide  easy  to  follow  assessments  of  cost 
impacts  to  decision  variables  and  present  a  range  of  costs  associated  with  each 
sustainment  option  with  standards  of  describing  cost  in  net  present  value  or  in  base-year 
fiscal  dollars.  The  next  BCA  should  leverage  available  databases  to  provide  routine 
updates,  concurrent  with  regularly  scheduled  inputs,  to  generate  a  Continuous  BCA. 
When  successfully  completed,  the  next  BCA  can  allow  USAF  senior  leaders  to: 

•  Analyze  alternatives  before  proceeding  with  further  sustainment  decisions 
regarding  the  C-17A. 

•  Create  projections  on  future  benefits,  costs,  and  risks  to  C-17A 
sustainment 
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•  Further  inform  the  five-year  development  plan  (FYDP)  and  program 
objeetive  memorandum  (POM)  to  align  budgeting  deeisions  with 
sustainment  objectives 

Recommendations  for  future  research  will  both  compliment  and  build  upon  this 
research  with  the  intent  to  provide  senior  leaders  with  the  necessary  tools  to  aid  in 
sustainment  decisions.  This  research  centered  around  informing  the  next  business  case 
analysis  and  to  that  extent,  more  research  needs  to  be  directed  in  completing  the  risk, 
cost,  and  incentive  models  and  to  delve  further  into  options  for  long-term  engine 
maintenance  and  support. 

Continuous  BCA 

Business  case  analyses  are  most  valuable  when  they  can  be  updated  and  compared 
to  previous  work  and  analysis.  Comparisons  can  identify  trends  and  expose  areas  for 
improvement  by  changing  inputs.  Current  guidance  requires  BCAs  be  performed  every 
five  years  to  determine  fiscal  and  sustainment  progress,  issues,  and  concerns  as  well  as 
serving  to  update  previous  assumptions  on  the  environment.  These  BCAs  are  costly  and 
when  not  inclusive  of  mutually  exclusive  metrics,  fail  to  capture  the  full  spectrum  of 
sustainment  courses  of  action.  A  recommendation  for  future  research  follows  four 
phases.  First,  the  risk  assessment  from  the  2009  BCA  must  be  updated  and  matched  to 
the  performance  benefits  suggested  by  this  research.  Next,  the  cost  metrics  must  be  re¬ 
assessed  on  the  appropriate  blend  of  government-contractor  activities  mix  for  next  BCA. 
Incentive  structures  should  be  studied  to  look  specifically  at  maintaining  the  current 
sustainment  activities  while  managing  the  contractor  differently  in  order  to  realize  the 
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required  cost  savings  directed  for  the  program.  Lastly,  a  Continuous  BCA  should 
develop  from  these  four  products:  cost  estimates,  risk  analysis,  and  incentive  program 
structures.  The  Continuous  BCA  should  leverage  existing  data  systems,  exporting 
information  when  routine  updates  are  made,  to  populate  an  interactive  display  allowing 
for  the  simultaneous  depiction  of  all  benefit,  cost,  and  risk  metrics  as  well  as  incentive 
changes  in  addition  to  a  future  projection  given  historical  data.  Comparative  analysis 
could  project  expenses  given  certain  detailed  incentive  programs  such  as  the  standard 
firm,  fixed  incentive  program  but  also  include  cost-plus.  The  Continuous  BCA  can  then 
be  used  to  inform  the  five-year  development  plan  (FYDP)  and  the  program  objective 
memorandum  (POM)  process.  The  Continuous  BCA  would  represent  an  additional  tool 
used  by  senior  leaders  to  evaluate  the  impact  of  performance,  cost,  and  risk  on 
sustainment  strategies. 

Competition  of  C-17A  Engine  Sustainment 

Further  analysis  should  assess  the  BCA’s  suggestion  to  compete  the  Pratt  & 
Whitney  FW-1 17  engines.  The  assumed  cost  savings  identified  was  $1.6B  though  this 
assertion  not  neither  supported  with  data  nor  accompanied  a  method  for  transition.  The 
largest  potential  for  cost  savings  contained  no  inclusion  of  costs  relating  to  the  workforce 
designed  to  determine  a  transfer  plan.  Consequently,  much  of  this  projected  savings  may 
no  longer  be  available.  In  line  with  previous  recommendations,  consider  other  cost- 
incentives  instead  of  seeing  an  alternate  source  of  sustainment  for  the  C-17A  engines. 
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Incentive  Analysis 

From  the  model,  the  current  sustainment  strategy  results  in  the  lowest  NMCS, 
NMCM,  NMCB,  and  Depot  Time  rates.  Changing  the  weights  to  favor  Depot  Time  up  to 
almost  50%  of  the  model’s  total  weight  is  the  only  manner  for  an  alternate  program  to  be 
as  competitive  as  the  C-17A’s  current  sustainment  strategy.  Additionally,  transitioning 
the  program  to  an  alternate,  potentially  less  productive  sustainment  strategy  may  result  in 
higher  aircraft  availability  sub-factor  rates.  The  lack  of  a  credible  alternative  sustainment 
strategy  makes  a  stronger  argument  for  managing  the  contractor  differently  through 
incentives. 

In  2007,  the  Center  for  Strategic  and  International  Studies  released  a  report 
intended  to  further  inform  the  discussion  of  cost-plus  contracts  in  the  DoD.  This  report 
offers  several  different  alternatives  to  cost-plus  incentive  structures.  A  full  BCA  may  not 
reveal  best-value  decisions  from  re-allocating  sustainment  activities.  The  mix  of 
Government  and  contractor  activities  should  also  be  contrasted  with  analysis  of 
incentives.  Currently,  the  C-17A  aircraft  availability  rates  are  higher  than  goal  standard. 
Transitioning  the  program  to  a  different  sustainment  strategy  may  incur  sub-standard 
performance  and  not  represent  areas  for  significant  savings.  However,  moving  away 
from  the  cost-plus  incentive  structure  may  represent  opportunities  for  savings  without 
negatively  impacting  aircraft  availability. 

Summary 

After  20  years  of  service,  the  C-17A  consistently  demonstrates  some  of  the 
highest  levels  of  aircraft  availability  of  any  aircraft  in  the  USAF  inventory.  A  2006 
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DoDIG  report  directed  the  creation  of  a  BCA  to  determine  whether  this  performance  was 
matched  to  best-value  analysis.  In  order  to  fully  inform  discussions  on  best-value 
sustainment  strategies  a  BCA  should  be  able  to  quantify  exceeding  aircraft  availability 
standards  means  in  terms  of  cost  and  risk.  Determining  the  appropriate  balance  of  trade 
offs  can  only  be  made  once  the  sustainment  decision  is  fully  informed.  A  thorough  and 
complete  BCA  is  the  one  of  the  most  important  steps  in  this  assessment  and  one  that 
needs  to  be  carefully  considered  and  created. 

The  2009  C-17A  BCA  recommendations  positively  affirmed  a  government-led 
collaboration  between  the  Air  Force  and  The  Boeing  Company  as  the  best-value  solution 
for  C-17  sustainment  (BCA  Executive  Summary,  2009).  This  recommendation  follows 
current  legal  requirements  set  forth  in  2008  by  the  United  States  Congress  requiring 
government-led  integration  support  (need  citation  of  this  law).  Furthermore,  the 
recommendations  also  positively  contributed  to  the  Air  Force’s  goal  of  maximized 
aircraft  availability  through  incentives  given  for  performance  and  the  reduction  of 
unavailability  based  on  maintenance,  supply  and  depot  issues. 

Overall,  the  2009  BCA  assessed  seven  potential  cases  for  consideration.  The 
cases  were  constructed  according  to  integrator  type  and  support  provider.  Furthermore, 
they  were  evaluated  according  to  their  ability  to  attain  certain  specific  and  measureable 
attributes  relating  to  depot  performance.  These  attributes  were  compiled  then  prioritized 
from  subject  matter  experts  examining  workload  assignments  relative  to  pre-defined 
criteria.  Following  the  analysis,  the  BCA  set  forth  three  courses  of  action,  recommending 
one  case  as  the  best  value  of  benefits,  cost,  and  risk.  From  the  recommendation  of  the 
BCA,  the  US  Air  Force,  acting  on  behalf  of  the  other  governments  in  partnership  with  the 


51 


C-17A  fleet,  made  several  ehanges  to  the  recommendation.  The  changed 
recommendation  was  then  presented  to  The  Boeing  Company  (hereafter  referred  to  as 
Boeing).  Boeing  offered  to  meet  the  cost  savings  requirements  without  transitioning  to 
the  recommended  course  of  action.  The  Boeing  offer  represented  the  status  quo  with 
continued  efforts  to  reduce  costs  and  realize  performance  improvements.  The  current 
state  of  affairs  notes  little  change  to  the  2009  environment  with  the  exception  of  the 
government  acting  as  the  primary  source  manager  with  the  addition  of  an  ongoing  effort 
to  transition  the  engine. 

This  project  developed  mutually  exclusive  BCA  benefit  metrics  to  assess  different 
sustainment  strategy  benefits  and  developed  weights  to  use  within  a  decision  analysis 
framework  to  better  inform  future  BCAs.  Depot  sustainment  strategies  were  compared 
using  mutually  exclusive  metrics.  This  comparison  was  made  to  determine  how  each 
weapons  system  addresses  sub-factors  of  aircraft  availability.  Business  rules  were 
applied  to  collected  data.  These  business  rules  excluded  data  unrepresentative  of  the 
sustainment  strategy  and  also  informed  future  transition  decisions.  Current  sustainment 
strategies  were  used  as  potential  options  for  transitioning  the  C-17A  program. 

The  current  C-17A  sustainment  strategy  appears  to  offer  the  best  benefit  when 
compared  to  the  alternatives  considered  in  this  project.  The  C-17A  has  the  lowest 
combination  of  sustainment  sub-factors.  Regardless  of  the  importance  placed  on  specific 
attributes,  or  the  value  of  specific  weights,  the  C-17A  out-performs  other  sustainment 
strategies  analyzed  in  the  benefit  model  overall.  The  cost,  risk,  and  incentive  analysis 
will  further  inform  whether  the  current  sustainment  strategy  is  the  best-value  decision  and 
may  reveal  potential  for  cost  savings. 
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Appendix  A:  List  of  Figures 


Statistical  Summaries 


NMCS  Rate 


Figure  3:  Percent  Non-Mission  Capable  Rate  for  Supply 
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Figure  4:  Percent  Non-Mission  Capable  Rate  for  Maintenance 
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NMCB  Rate 
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Figure  5:  Percent  Non-Mission  Capable  Rate  for  Both  Supply  and  Maintenance 
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Figure  6:  Percent  Depot  Time  Rate 
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Excluded  Data 


C-17ANMCS/M/B:  FY98toFY13 


Figure  7:  C-17A  NMCS/M/B  &  Depot  Time  Rates 


C-5M  NMCS/M/B:  FY09toFY13 


Figure  8:  C-5M  NMCS/M/B  &  Depot  Time  Rates 
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C-130E  NMCS/M/B:  FY94toFY13 


Figure  9:  C-130  E  NMCS/M/B  &  Depot  Time  Rates 


E-8C  NMCS/M/B:  FY96  to  FY13 


Figure  10:  E-8C  NMCS/M/B  &  Depot  Time  Rates 
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C-130J  NMCS/M/B:  FY04  to  FY13 


Figure  11:  C-130J  NMCS/M/B  &  Depot  Time  Rates  (shaded  region  shows  Business 
Rules  applied  though  none  of  this  program  data  was  used  in  the  final  analysis). 
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Appendix  B:  List  of  Tables 


Table  3:  List  of  Acronyms  and  Definitions 


NMCS  -  Non-Mission 

Capable  for  Supply 

Percentage  of  aircraft  that  cannot  perform  any  assigned  missions  because  of  a 
supply  condition(s). 

NMCM  -  Non-Mission 
Capable  for  Maintenance 

Percentage  of  aircraft  that  cannot  perform  any  assigned  missions  because  of  a 
maintenance  condition(s). 

NMCB  -  Non-Mission 

Capable  for  Both 

Percentage  of  aircraft  that  cannot  perform  any  assigned  missions  because  of  a 
supply  and  maintenance  condition(s). 

UPRN  -  Unit  Possessed  Not 
Reported 

Percentage  of  aircraft/equipment  that  is  unit  possessed  not  reported  is 
downtime  when  the  unit  still  possesses  the  equipment  but  is  not  charged  with 
the  downtime. 

TAI  -  Total  Aircraft 

Inventory 

The  total  number  of  aircraft  possessed  in  the  Air  Force  Inventory  for  that 
particular  model  type  and  designation. 

Depot  Rate 

Percent  of  aircraft/equipment  possessed  by  a  depot  (govemment/contractor 
facility)  or  depot  field  team. 

Table  4:  List  of  Formulas 


NMCS  Rate: 


NMCS  hours  ^ 
TAl  hours 


NMCM  Rate: 


NMCM  hours  , 

- = - X  100 

TAI  hours 


NMCB  Rate: 


NMCB  hours  . . . 

- = - X  100 

TAl  hours 


Depot  Time  Rate: 


Depot  hours  , 

— - — = - X 100 

TAI  hours 


TAI  Rate: 


UPRN  Rate: 


TAI 

TAI  _  possessed  _  hours 


xlOO 


UPNR  hours  ^ 
TAI  hours 


PDQR  Rate: 


Deficient  Parts 

— - - = - xlOO 

TAI  hours 
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Table  5:  Current  Performance  Standards  and  Observations  (June,  2013) 


C-17A  Metrics:  2013  Performance  vs. 

Standards 

Available  (%)  Projection 

74.80% 

Available  (%)  Standard 

72.90% 

Available  (%) 

74.14% 

Depot  (%)  Projection 

14.80% 

Depot  (%) 

15.05% 

UPNR  (%)  Projection 

0.80% 

UPNR(%) 

0.51% 

NMCS(NA)  (%)  Projection 

1.50% 

NMCS(NA)  (%) 

1.46% 

NMCM(NA)  (%)  Projection 

9.30% 

NMCM(NA)  (%) 

7.74% 

NMCB(NA)  (%)  Projection 

1.20% 

NMCB(NA)  (%) 

1.11% 

MC  (%)  standard 

87.50% 

TNMCM(%)  standard 

9.40% 

TNMCS  (%)  standard 

7.00% 
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